Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Investigating the role of muscle physiology and spinal circuitry in sensorimotor control
(USC Thesis Other)
Investigating the role of muscle physiology and spinal circuitry in sensorimotor control
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
INVESTIGATING THE ROLE OF MUSCLE PHYSIOLOGY AND SPINAL CIRCUITRY IN SENSORIMOTOR CONTROL by George A. Tsianos A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (BIOMEDICAL ENGINEERING) December 2012 Copyright 2012 George A. Tsianos ii To my family. iii Acknowledgements I thank Dr. Gerald Loeb for investing the time and effort to help me grow as a researcher and as a professional. His invaluable guidance over the years was instrumental both toward my profound appreciation for quality research and my approaches to seeking insight into key questions in the field of motor control. He so willfully and effectively taught me knowledge accumulated from years of experimentation and analysis that was crucial for the projects I undertook. His teachings facilitated the development of critical and analytical thinking skills that I will cherish and use for a lifetime. I would also like to thank deeply the rest of my dissertation committee (Drs. Terence B. Sanger, Stefan Schaal, Vasilis Z. Marmarelis, Francisco Valero-Cuevas) for their constructive feedback that helped me develop my thinking further and enhance my research. Researching in the Medical Device Development Facility (MDDF) gave me the unique opportunity to work alongside people with great personalities and bright minds. I am grateful to my colleague and friend Giby Raphael, who helped me get acquainted with the literature and modeling of spinal cord physiology and to professor Rahman Davoodi for his useful comments on my research throughout my doctoral studies. Jeremy Fishel and Zhe Su from the tactile sensor group had a very iv positive influence on my research as their innovative and intriguing work constantly motivated me to excel. Norman Li, Jared Goodner, and John Sunwoo from the SLR group assisted with implementing models of muscle physiology, spinal circuitry, and motor learning. I thank the rest of the MDDF group for their support and friendship over the years that made my doctoral studies overall a pleasant and valuable learning experience. I am forever grateful to my parents, Peter and Pandora Tsianos, for their love, support and faith in my abilities that gave me the strength to deal with the challenges of research. I also thank them for setting the example of hard work and perseverance that had a key role in shaping my character and fueling the progress of this research. My older brother, Niko Tsianos, and my sister-in-law, Elena, made this possible not only through their constant encouragement to dream big but also by setting the example of good morals and optimism that I believe are crucial for striving to perform rigorous research that benefits society. I would also like to express my deep gratitude to George Jaber, a great friend whose diligence and devotion to knowledge kept me motivated throughout my studies at USC. I have been fortunate and very thankful to have a large number of extended family members and friends that were enthusiastic and appreciated the goal I had set. I especially thank Andria Shakolas for believing in me, for her patience, understanding, and for always being by my side to help me overcome any hurdles that came along the way. The countless hours I have spent discussing my research v with her and rehearsing talks have helped mature my thinking and clarify many difficult concepts. This research was made possible by funding from the Myronis Fellowship Foundation and the DARPA REPAIR program. vi Table of Contents Acknowledgements .............................................................................................................................. iii List of Tables ............................................................................................................................................. x List of Figures .......................................................................................................................................... xi Abstract ....................................................................................................................................................... 1 Chapter 1: Introduction ........................................................................................................................ 3 Motivation ........................................................................................................................................... 3 Main contributions .......................................................................................................................... 5 Organization of the Thesis ............................................................................................................ 6 Chapter 2: Muscle Physiology and Modeling ................................................................................ 8 Preface .................................................................................................................................................. 8 Contributions of the Authors ................................................................................................ 8 Introduction ....................................................................................................................................... 8 Relevant physiology .............................................................................................................. 10 Structure of the model ................................................................................................................ 12 Active force ............................................................................................................................... 14 Passive elastic elements ...................................................................................................... 31 Energetics ................................................................................................................................. 38 Balancing accuracy and computational efficiency ........................................................... 45 Parameters needed to model specific muscles .................................................................. 46 Optimal fascicle length (L 0) ................................................................................................ 46 Muscle mass (m) .................................................................................................................... 47 Fiber composition .................................................................................................................. 48 Characteristic firing rate of muscle fiber type (f 0.5).................................................. 50 Tendon + aponeurosis length (L 0 T ) ................................................................................. 50 Maximum musculotendon path length (L max MT ) ........................................................ 51 Pennation angle ...................................................................................................................... 51 Chapter 3: Mammalian Muscle Model for Predicting Force and Energetics during Physiological Behaviors ................................................................................................... 52 Preface ............................................................................................................................................... 52 Abstract ............................................................................................................................................. 53 Introduction .................................................................................................................................... 54 vii Methods ............................................................................................................................................ 58 Model Design and Rationale .............................................................................................. 58 Implementation of the Model ............................................................................................ 62 Overall Behavior of the Model .......................................................................................... 70 Validation ......................................................................................................................................... 76 Effects of Fiber Composition ............................................................................................. 76 Effects of Excitatory Drive ......................................................................................................... 78 Effects of Kinematics ............................................................................................................ 80 Discussion ........................................................................................................................................ 83 Limitations ............................................................................................................................... 83 Role in Motor Control Research ....................................................................................... 86 Role in Studying Effort and Fatigue ................................................................................ 87 Appendix A: Virtual Muscle ‘Lumped Units’ Algorithm .......................................... 89 Appendix B: Modeling details for dynamic knee extension task ......................... 99 Acknowledgment ................................................................................................................ 107 Chapter 4: Spinal Circuitry Physiology and Modeling ........................................................ 108 Neurophysiology ........................................................................................................................ 108 Interneurons identified via electrophysiology ............................................................... 109 Other methods to identify interneuronal circuits ......................................................... 114 Chapter 5: Previously Developed Models of Components ................................................ 118 Muscle force and energetics .................................................................................................. 119 Muscle spindle ............................................................................................................................. 122 Golgi tendon organ .................................................................................................................... 124 Spinal cord .................................................................................................................................... 125 Chapter 6: The Relationship between the Spinal Cord and other Levels of the Sensorimotor System .................................................................................................................... 127 Chapter 7: Spinal-Like Regulator Facilitates Control of a Two Degree-of- freedom Wrist .................................................................................................................................. 131 Preface ............................................................................................................................................ 131 Contributions of the Authors .......................................................................................... 132 Abstract .......................................................................................................................................... 132 Introduction ................................................................................................................................. 133 Methods ......................................................................................................................................... 135 Biomechanical model ........................................................................................................ 135 Results ............................................................................................................................................ 149 Stabilizing response to external force perturbation ............................................. 149 Rapid voluntary movement to a position target ..................................................... 151 Voluntary isometric force to a target level ............................................................... 152 Adaptation to viscous curl force fields ....................................................................... 154 Multiple Local Minima ...................................................................................................... 159 viii Learning Rate ....................................................................................................................... 161 Comparison with Servo-Control ................................................................................... 162 Sensitivity of optimized solutions ................................................................................ 165 Generalization of optimized solutions ........................................................................ 168 Discussion ..................................................................................................................................... 170 Limitations of the Model .................................................................................................. 170 The General Utility of Regulators ................................................................................. 171 Interpretation of Deafferentation Experiments ...................................................... 173 Hierarchical Architecture ................................................................................................ 175 Acknowledgments ..................................................................................................................... 178 Chapter 8: Modeling the Potentiality of Spinal-Like Circuitry for Stabilization of a Planar Arm System ................................................................................................................ 179 Preface ............................................................................................................................................ 179 Author contributions ......................................................................................................... 180 Abstract .......................................................................................................................................... 180 Introduction ................................................................................................................................. 181 Methods ......................................................................................................................................... 185 Simulation environment .................................................................................................. 185 Musculoskeletal model ..................................................................................................... 186 Proprioceptor models ....................................................................................................... 187 Spinal cord model ............................................................................................................... 188 Brain model ........................................................................................................................... 191 Modeled task ......................................................................................................................... 191 Results ............................................................................................................................................ 193 Discussion ..................................................................................................................................... 196 Role of co-contraction in learning novel tasks ............................................................... 196 Role of the spinal cord.............................................................................................................. 198 Limitations .................................................................................................................................... 198 Implications for BMIs ............................................................................................................... 200 Chapter 9: A Realistic Model of Spinal Circuitry Facilitates Control of Center- out Reaching Movement .............................................................................................................. 201 Preface ............................................................................................................................................ 201 Author contributions ......................................................................................................... 202 Introduction ................................................................................................................................. 202 Methods ......................................................................................................................................... 206 Musculoskeletal system.................................................................................................... 206 Model of spinal circuitry .................................................................................................. 207 Model of the neuron ........................................................................................................... 208 Firing rate versus explicit spikes .................................................................................. 210 Results ............................................................................................................................................ 219 Exemplary solution to a reaching task .............................................................................. 219 ix Learning curves ................................................................................................................... 221 Variability of performance .............................................................................................. 222 Effects of energetics on learning ................................................................................... 225 Learning to reach in other directions ......................................................................... 225 Kinematics and kinetics validation .............................................................................. 227 Timing and magnitude of initial agonist activity .................................................... 234 Timing and magnitude of initial activity of individual muscles ........................ 239 Discussion ..................................................................................................................................... 244 Temporal structure of descending commands........................................................ 244 Solution space afforded by the lower motor system ............................................ 246 Effects of performance criteria on learning .............................................................. 249 Which aspects of spinal circuitry contribute to its intrinsic properties? ...... 250 Limitations of the learning algorithm ......................................................................... 251 Chapter 10: Motor Programs to the Spinal-Like Regulator are Interpolable ............ 253 Preface ............................................................................................................................................ 253 Author contributions ......................................................................................................... 254 Introduction ................................................................................................................................. 254 Methods ......................................................................................................................................... 257 Model of the lower motor system................................................................................. 257 Model of descending commands ................................................................................... 257 Learning algorithm ............................................................................................................. 258 Training sequence .............................................................................................................. 258 Interpolation method ........................................................................................................ 261 Results ............................................................................................................................................ 263 Discussion ..................................................................................................................................... 276 Chapter 11: Conclusions and Future Directions ................................................................... 279 Conclusions .................................................................................................................................. 279 Future Directions ....................................................................................................................... 280 Intelligent coordinate descent ....................................................................................... 280 A physiologically plausible learning model .............................................................. 281 Extending the model to a 3D shoulder-elbow system .......................................... 284 Driving the SLR with cortical activity ......................................................................... 285 Regulators for different classes of systems .............................................................. 287 References ............................................................................................................................................ 289 x List of Tables Table 3-1: Table of Equations ....................................................................................................... 101 Table 3-2: Symbols and Definitions ........................................................................................... 103 Table 9-1: Symbols and definitions ............................................................................................ 216 Table 10-1: Interpolation test summary .................................................................................. 260 xi List of Figures Figure 2-1: Model Overview ........................................................................................................... 13 Figure 2-2: Motor Unit Recruitment and Modulation. ........................................................... 15 Figure 2-3: Calcium Kinetics and Cross-bridge Activation .................................................. 17 Figure 2-4: Sag ....................................................................................................................................... 19 Figure 2-5: Force-Length relationship ......................................................................................... 21 Figure 2-6: Force-Velocity relationship ...................................................................................... 23 Figure 2-7: Length Dependence of Cross-bridge Activation ............................................... 25 Figure 2-8: Length Dependence of Force-Velocity. ................................................................. 29 Figure 2-9: Yielding ............................................................................................................................. 30 Figure 2-10: Parallel Elastic Force................................................................................................. 32 Figure 2-11: Thick Filament Compression ................................................................................. 35 Figure 2-12: Series Elastic Tendon/Aponeurosis Force ....................................................... 38 Figure 2-13: Partitioning of Energy Driving Muscle Contraction...................................... 42 Figure 3-1: Overview of the energetics algorithm. ................................................................. 69 Figure 3-2: Model verification ......................................................................................................... 71 Figure 3-3: Overall behavior of the model ................................................................................. 73 Figure 3-4: Comparison of force production and energy consumption ......................... 75 Figure 3-5: Validation results for effects of fiber composition .......................................... 78 Figure 3-6: Validation results for effects of excitatory drive .............................................. 80 Figure 3-7: Validation results for effects of kinematics ........................................................ 82 xii Figure 3-8: Comparison of ‘Natural’ and ‘Lumped’ recruitment schemes (a) .............. 93 Figure 3-9: Comparison of steady state response to an excitatory step ......................... 95 Figure 3-10: Comparison of dynamic response to sinusoidal input ................................ 96 Figure 3-11: Muscle model overview ........................................................................................... 97 Figure 3-12: Model of the dynamic knee extension task ................................................... 106 Figure 3-13: Characterization of the ATP/PCr recovery system .................................... 107 Figure 4-1: Classical interneuronal pathways ....................................................................... 111 Figure 5-1: The schematic on the left depicts the various components of the muscle model and their interactions (Song et al. 2008) ................................................. 122 Figure 5-2: Spindle model .............................................................................................................. 123 Figure 5-3: Golgi tendon organ model ...................................................................................... 125 Figure 7-1: Simulation environment............................................................................................. 137 Figure 7-2: Spinal circuitry between true-antagonists ...................................................... 139 Figure 7-3: Spinal circuitry between partial-synergists .................................................... 140 Figure 7-4: Stabilizing response to external force perturbation .................................... 150 Figure 7-5: Voluntary Isometric force to a target level ...................................................... 154 Figure 7-6: Adaptation to viscous curl force fields .............................................................. 156 Figure 7-7: Analysis of gain values ............................................................................................. 158 Figure 7-8: Typical learning curve for each task .................................................................. 162 Figure 7-9: Servo-control model ................................................................................................. 164 Figure 7-10: Sensitivity analysis ................................................................................................. 167 Figure 7-11: Ability of SLR strategy to generalize to new task conditions ................. 169 Figure 7-12: Generalizing the concept of hierarchical control ........................................ 177 xiii Figure 8-1: Schematic overview of the neuromusculoskeletal system of the planar arm ......................................................................................................................................... 185 Figure 8-2: Musculoskeletal system parameters .................................................................. 186 Figure 8-3: Distribution of gains among the various interneurons and muscles ............................................................................................................................................... 190 Figure 8-4: Overview of the modeled task .............................................................................. 192 Figure 8-5: Learning curves with and without cocontraction ......................................... 194 Figure 8-6: Learning curves for systematic application and subsequent removal of cocontraction ............................................................................................................ 196 Figure 9-1: Five classical interneuronal pathways that comprise the model from the perspective of a single muscle ................................................................................ 208 Figure 9-2: Overview of the neuron model ............................................................................. 210 Figure 9-3: Exemplary reach to the outward-left target. ................................................... 220 Figure 9-4: Exemplary learning curve ...................................................................................... 222 Figure 9-5: Multiple learning curves and the effects of energetics on learning ....... 224 Figure 9-6: Center-out reaches to multiple directions ....................................................... 227 Figure 9-7: Comparison of experimental and modeled kinematics .............................. 230 Figure 9-8: Comparison of experimental and modeled kinetics..................................... 233 Figure 9-9: Comparison of agonist initial activity between experiment and model .................................................................................................................................................. 238 Figure 9-10: Comparison of onset of muscle activity between experiment and model .................................................................................................................................................. 241 Figure 9-11: Comparison of magnitude of initial muscle activity between experiment and model ................................................................................................................. 244 Figure 9-12: Solution space analysis ......................................................................................... 248 Figure 10-1: Interpolation results for reach direction – training sequence 1 ........... 265 Figure 10-2: Interpolation results for reach direction – training sequence 2 ........... 266 xiv Figure 10-3: Interpolation results for reach direction – training sequence 3 ........... 267 Figure 10-4: Interpolation results for reaching distance. ................................................. 268 Figure 10-5: Interpolation results for reaching duration .................................................. 270 Figure 10-6: Interpolation results for curl-field magnitude............................................ 273 Figure 10-7: Interpolation results using reaches spaced 90 o apart .............................. 274 Figure 10-8: Effects of location of learned solutions in the solution space ................ 276 Figure 11-1: BioSearch overview ................................................................................................ 283 1 Abstract George A. Tsianos Making voluntary movements requires proper recruitment of muscles that exert torques at the joints. The brain controls these torques indirectly by sending commands to the spinal circuitry, which continuously integrates them with proprioceptive feedback and recurrent projections from motoneurons. The nature of this transformation and its implications for motor control have been investigated by building a realistic model of the lower motor system (spinal circuitry plus musculoskeletal system) and determining how much of the dynamics of reaching movement it can generate entirely on its own. An oversimplified model of the brain was employed in order to force the lower motor system to generate all of the necessary dynamics. Its outputs, representing supraspinal control of fusimotor gain, interneuronal biasing activity and presynaptic inhibition/facilitation, controlled a realistic set of spinal circuits based on the classical interneuronal types (propriospinal, monosynaptic Ia-excitatory, reciprocal Ia-inhibitory, Renshaw inhibitory and Ib-inhibitory pathways). Commands to the spinal circuitry were unmodulated step functions whose amplitudes were trained using a simple optimization algorithm and a cost function. 2 Despite the large number of control points in the spinal cord model (greater than 400 for our six-muscle model) and the oversimplified descending inputs, it was surprisingly easy to train the system to perform motor tasks such as resisting an impulsive perturbation applied to the endpoint and center-out reaches to multiple directions along with the complex muscle dynamics required to achieve them. This is because the high-dimensional space to be controlled appears to have many “good enough” solutions and relatively few undesirable local minima. Initially, energetics were not part of the performance criteria; nevertheless, the emerging strategies of muscle recruitment in those solutions were often metabolically efficient. Incorporating energetics into the cost function further improved the efficiency while maintaining acceptable kinematic behavior. It was also shown that solutions to new tasks (e.g. having untrained durations directions and distances) can be interpolated from solutions to tasks learned previously. Such generalization of solutions improves the rate of learning in many situations and also reduces the storage capacity required for the brain to memorize motor repertoires. These results suggest that the genetically specified circuitry of the spinal cord may have evolved to permit rapid and reliable solutions to new sensorimotor problems and that it tends to facilitate solutions with low energetic cost. These properties of spinal circuitry must be considered when assigning functionality to the motor control centers of the brain. 3 Chapter 1: Introduction George A. Tsianos Motivation We have the remarkable ability to make a wide range of movements that are essential to our daily lives, although it often goes unnoticed due to the ease at which we perform them. The neural processes that underlie even the simplest movements are highly complex and poorly understood. An improved understanding of the neural mechanism would enable clinicians to make a better association between motor disorder symptoms and their causes, thus improving diagnosis and enabling precisely targeted treatments. Additionally, revealing the principles that enable the neural control of movement would inform the design of robots that move more naturally and better cope with the control challenges posed by unstructured environments. The purpose of the research described here was to improve our knowledge of the emergent properties of muscle and spinal circuitry physiology that the brain must interact with to control motion of the limbs. The complexity of making a specific movement becomes evident after injury when we are faced with the challenge of learning to adjust our learned behavior to avoid discomfort or pain. When a ligament in the ankle is injured, for example, it is 4 important to restrict its range of motion for a critical time period. Determining the orientation of the ankle that causes discomfort is simple, but determining the appropriate changes in torque at the joints during walking that avoid that orientation is difficult. Furthermore, we do not have torque motors at our joints; instead, the torques are produced by muscles depending on the contractile force that they generate and the moment arm of the force with respect to the joint axis of rotation. The moment arm is defined by the muscle’s path between the places on the skeleton to which it attaches, and therefore, changes as a function of angle. When joint angle changes, the force producing capability of muscles also changes so, unlike torque motors, the same excitation input to the muscle will generate different levels of force depending on the motion itself. The intrinsic properties of the musculoskeletal system, therefore, determine the appropriate muscle excitation patterns that will accomplish the desired movement. To make matters even more complicated, the brain generally does not have direct control of muscles. Instead, it commands interneurons in the spinal cord that continuously mix these signals with signals from somatosensory receptors located in the periphery. Hence, the actual control problem that the brain is faced with is neither to determine the torques applied at the joints nor the contractile forces generated by muscles, but to determine the inputs to interneurons which will generate the desired movement. Computing the necessary changes in interneuronal input that would realize a desired movement is an extremely difficult task, yet we are all able to learn them quickly with a modest amount of practice. 5 The chapters that follow present a review of the physiology of the lower motor system (spinal circuitry plus musculoskeletal system) as well as detailed models and simulation experiments that provide new insight into how the brain learns to control complicated motor tasks rapidly. Main contributions • Development of a validated model of metabolic energy consumption at the muscle level • Development of a computationally efficient model of motor unit recruitment that improved the computational efficiency of a validated model of force production (Virtual Muscle) approximately 10-fold • Design of a general model of spinal circuitry that applies to a wide range of musculoskeletal systems • Demonstration of favorable intrinsic properties of a model of the lower motor system (spinal circuitry plus musculoskeletal system) of the arm. The following intrinsic properties were identified: ability to generate all of the dynamics of center-out reaching movement, solution space afforded by the lower motor system has a high density of good-enough local minima, command strategies controlling the lower motor system generalize. 6 Organization of the Thesis Chapter 2 reviews mammalian muscle physiology and modeling. It reviews the experiments conducted to characterize the major physiological processes underlying muscle contraction and describes how the results were integrated to create the validated computational model of muscle force production and energy consumption used in this research. Chapter 3 describes the design and validation of a model of metabolic energy consumption at the muscle level. It also presents a new motor unit recruitment algorithm that lumps all motor units within a fiber type into one mathematical entity, thereby improving the computational efficiency of the muscle model substantially. Chapter 4 reviews the literature on the connectivity of spinal circuits and the various techniques for identifying them. The specific set of circuits employed in the neuromusculoskeletal models in this study as well as justification for the circuits that were omitted is discussed. Chapter 5 reviews previously developed models of muscle physiology and spinal circuitry. Chapter 6 describes the motor control problem faced by the brain given the intrinsic properties of the spinal circuitry and musculoskeletal mechanics that it interacts with. Results from several studies that provide important insights are discussed. 7 Chapter 7 introduces a model of spinal circuitry (spinal-like regulator; SLR) for the wrist musculoskeletal system and demonstrates that, despite having a high- dimensional control space, it facilitates control of a wide range of tasks. The solution space afforded by the SLR appears to contain a large number of good- enough local minima that are easily discoverable even when descending inputs are constrained to possess no dynamics. Chapter 8 introduces an SLR for a planar elbow-shoulder musculoskeletal system and its intrinsic properties are studied in the context of a stabilization task. Chapter 9 demonstrates that the SLR facilitates control of center-out reaching movement despite the kinetic redundancy and interaction torques inherent to this system. The muscle activation patterns that emerge are similar to those observed experimentally. Chapter 10 shows that solutions to the SLR for center-out reaching tasks interpolate for a wide range of reaching parameters, which reduces both the memory needed for storing motor repertoires as well as the time needed to learn new ones. Chapter 11 outlines the major conclusions of the modeling studies and provides an overview of the future directions that are enabled and/or motivated by this work. 8 Chapter 2: Muscle Physiology and Modeling George A. Tsianos, Gerald E. Loeb Preface The manuscript presented in this chapter is scheduled to appear in Scholarpedia, an online encyclopedia that is peer-reviewed. It reviews the major physiological processes underlying muscle contraction and how they were characterized and implemented into the muscle model used in this research. Contributions of the Authors George A. Tsianos wrote the manuscript. Gerald E. Loeb edited the manuscript. Introduction Quantitative models of muscle contraction are crucial for understanding neural control of movement. The nervous system must coordinate commands to a large set of muscles in a precise sequence. Characterizing their force producing capabilities provides insight into the feasible set of muscle activation patterns that lead to a desired behavior. The actual strategy selected from this set will likely depend on other factors such as metabolic energy consumption, whose minimization is clearly advantageous for survival. Both force generation and energy expenditure depend 9 complexly on both the commands from the nervous system and the kinematics of muscle length and velocity. Computational models that relate these relationships to architectural components of muscle provide a means to validate theories of muscle physiology. Computational models that accurately capture these relationships are an essential component of complete models of musculoskeletal systems (see Musculoskeletal Mechanics and Modeling). Neural control of muscles arises from a combination of commands from the brain as well as feedback from the periphery that are integrated by interneuronal circuits in the spinal cord. Muscle itself is a major source of the peripheral feedback, as it possesses many specialized mechanoreceptors that are sensitive to stretch and tension (see Proprioceptors and Transduction Modeling). Both the stretch and tension experienced by muscle depend on the motion of the skeletal segments to which they attach; for a detailed explanation of the interactions between muscles, the skeleton, and environmental objects, see Musculoskeletal Mechanics and Modeling. The physiological properties of muscle can change over time. In the short term, force generation may increase or decrease as a result of chemical changes associated with potentiation and fatigue, respectively. In the longer term, muscle morphology and physiology may change as a result of trophic responses to patterns of use. All of these effects tend to be specific to the various muscle fiber types, making it important to develop muscle models that reflect the subpopulations of fiber types and the relative recruitment of the motoneurons that control them. 10 Relevant physiology Each muscle is controlled by a group of motoneurons known as a motor pool or motor nucleus. All motoneurons in such a pool generally receive the same drive signals, although there are some exceptions (Loeb and Richmond 1989). Each motoneuron along with all of its muscle fibers is defined as a motor unit, whose recruitment and firing rate in response to the same drive varies substantially depending on the size and impedance of the motoneuron (see Frequency- recuitment). The force generated by a given motor unit at a given rate of firing depends on the total cross-sectional area of the muscle fibers in the unit. Larger motoneurons tend to innervate larger numbers of larger diameter muscle fibers. The force production and corresponding energy consumption of each motor unit depends mainly on its length, velocity and firing rate. Because muscle fibers within a motor unit generally have the same contractile properties, they can be lumped into one mathematical entity whose behavior (force production and energy consumption) is proportional to the total cross-sectional area of its fibers. All of the muscle fibers in all of the motor units of a given muscle tend to move together, experiencing the same sarcomere lengths and velocities. Because of this homogeneity, most of the experimental phenomena related to contraction of whole muscle can be explained by processes occurring at the sarcomere level. The sarcomere is the basic unit of the contractile apparatus. It is demarcated at its ends by thin Z-plates from which a matrix of thin filaments of actin project in each direction. These interdigitate with a matrix of thick filaments of myosin that 11 are held in the center of the sarcomere by strands of the highly elastic connectin filaments that tether the thick filaments to the Z-plates. In order for muscles to contract, protrusions from myosin called myosin heads must first be cocked to a relatively high-strain configuration and then attached to neighboring binding sites on the actin. The resulting cross-bridges act like springs that pull on the actin. The metabolic energy required to cock the myosin head is provided directly by ATP molecules that are present in the sarcoplasm. In order for myosin heads to attach to neighboring binding sites, a regulatory protein called tropomyosin that normally occludes actin binding sites must undergo a conformational change. This occurs indirectly through binding of calcium to troponin, a relatively smaller molecule that is bound to tropomyosin at regular intervals along its length. When calcium binds to troponin, a local conformational change is induced that exposes nearby actin binding sites so that the cocked myosin heads can attach to form cross-bridges. The normally relaxed state of inactive muscle is achieved by ATP-powered pumping of the calcium out of the sarcoplasm and into a network of vesicles called longitudinal tubules, preventing cross-bridge formation. When the muscle is activated, calcium is released from cisterns in these longitudinal tubules in response to action potentials elicited in the muscle fibers as a result of a chemical synapse with the motor axons. These action potentials propagate along the cell membrane and its invaginations deep into the muscle fiber called transverse tubules. The ATP molecules that cock the myosin heads and drive the ion pumps must be replenished eventually via catabolism of glucose occurring both in the sarcoplasm (glycolytic) 12 and within mitochondria (oxidative), consuming additional energy. The model of muscle presented here attempts to represent each of these structures and processes as explicit terms in the set of equations that comprise the model. Structure of the model The model is an assembly of sub-models, each characterizing a major physiological process underlying muscle contraction (Figure 1). Modeling physiological processes independently as opposed to the aggregate behavior avoids overfitting data to specific preparations and therefore improves the likelihood that the model will be valid under untested conditions. Moreover, the one-to-one correspondence of the model’s terms and coefficients to physiological processes makes it relatively straightforward to extend it to include unaccounted aspects of muscle contraction and to adjust its coefficients to model naturally occurring changes over time such as potentiation, exercise, fatigue and injury. The data driving the model originate from experiments on mammalian muscle that were designed to isolate specific processes that were then quantified. These include a wide variety of preparations with various muscle fiber architectures, but the model parameters have been normalized to dimensionless variables that depend on the constant structures of the sarcomeres, which are highly similar across all mammalian skeletal muscles. Independently modeled physiological processes and their and energy consumption. Figure 2-1: Model Overview Independently modeled physiological processes and their contribution to muscle force production 13 contribution to muscle force production 14 Active force Frequency-recruitment The drive, or synaptic current, to each motoneuron in a muscle’s motor pool is roughly the same, however, the firing rates that result in each motoneuron can be substantially different. This is because the action potentials that result in each axon depend on the change in the cell body’s membrane potential rather than the input current directly. The membrane potential induced by the input current depends on cell resistivity, which is larger for smaller motoneurons that innervate fewer muscle fibers. For this reason, as the drive increases, small motor units are recruited first, followed by larger motor units having higher recruitment thresholds (Henneman and Mendell 1981). Increasing the drive past the threshold of a given motoneuron results in higher firing rates until the drive saturates, at which point firing rate also saturates. This monotonic relationship can be approximated by a line whose slope depends on the recruitment threshold of the motor unit (see Figure 2). U f f U f f th MU env i i * 1 min max min − − + = Note that the firing rate is normalized to f 0.5 (frequency at which the motor unit produces half of its maximal isometric force) because motor units tend to fire between 1/2f 0.5 and 2f 0.5, the range over which muscle activation is most steeply modulated (see Calcium Kinetics below). 15 Figure 2-2: Motor Unit Recruitment and Modulation. The plot (top) emphasizes the relatively higher recruitment threshold of larger motor units (e.g. fast versus slow-twitch) and their firing rate modulation until the common drive (U) saturates the firing rate of all motor units. The schematic (bottom) provides a mechanistic explanation for this phenomenon, namely the size principle. The same synaptic current produces a smaller excitatory post synaptic potential (EPSP) in the larger motor neuron because its input impedance is lower. Therefore, larger input currents are necessary for overcoming the threshold for action potential generation and motor unit recruitment. Bottom figure source: Kandel ER, Schwartz JH, Jessel TM. Principles of Neural Science. 4th ed. New York (NY): McGraw-Hill; c2000. Figures 34-11; p.687. 16 Calcium kinetics and cross-bridge activation (f eff , Af) The sigmoidal relationship between motoneuron firing rate and activation of the contractile apparatus arises from the release, diffusion and reuptake of calcium (see Figure 3). At low pulse rates, the calcium released by each pulse is completely cleared before the next pulse, resulting in small, discrete twitches. As pulse rate increases, the calcium released by each pulse starts to accumulate to higher concentrations, allowing the calcium to diffuse further and expose more cross- bridge binding sites. Eventually, the calcium concentration in all parts of the muscle fiber become sufficient to expose all binding sites and activation plateaus even if firing rate continues to increase. This is called a tetanic contraction. 17 Figure 2-3: Calcium Kinetics and Cross-bridge Activation The effects of firing rate on isometric force output of muscle is shown from zero to tetanic activation. Overlaid graphics highlight underlying processes at the sarcomere level that give rise to the behavior. Only a small portion of the sarcomere is shown here due to symmetry. Thick horizontal bars represent thick filaments and thin horizontal lines represent thin filaments. Vertical bars correspond to Z-disks. The small red spheres are calcium ions whose bond with actin sites is indicated by short black lines that link them and a small red dot overlaid on top of the actin (grey dots indicate inactive binding sites). If a cross-bridge is formed then the small ovals in the figure representing myosin heads are colored blue and are in a cocked configuration (large angle with respect to their neck region). The equation below captures the effects of firing rate on cross-bridge activation. Y corresponds to yielding behavior exhibited by slow-twitch fibers and Sag observed in fast-twitch fibers. Both phenomena affect cross-bridge activation through hypothesized mechanisms discussed in the corresponding sections. Note that equation parameter n f is length dependent (see Length dependency of calcium kinetics). The shape of the sigmoid relationship is defined by a f, n f0, and n f1constants, 18 some of which are fiber type dependent. See Tsianos et al. 2012 for a complete definition of all model parameters (Tsianos, Rustin et al. 2012). ( ) − − = f n f f eff ce ce eff n a YSf V L f Af exp 1 , , − + = 1 1 1 0 ce f f f L n n n Sag (S) When fast-twitch muscle is excited isometrically with a constant drive, its force output increases initially to some maximal level and then decreases gradually (Figure 4). This is thought to occur due to an increase in the rate of calcium reuptake, which effectively reduces the concentration of calcium in the sarcoplasm and hence the number of crossbridges that can form (see review of physiology). 19 Figure 2-4: Sag Experimental data depicting the sag phenomenon in response to constant stimulation for several frequencies. The graphics on the bottom portion illustrate the effect of increasing the rate of calcium reuptake on calcium concentration and cross-bridge formation. This phenomenon can be modeled by a time-varying scaling factor applied to f eff, which is directly related to the calcium present in the sarcoplasm (see Calcium kinetics; Brown and Loeb 2000). ( ) ( ) S S eff T t S a f t S − = , & Force-length (FL) The amount of active force a muscle produces during maximal isometric contractions (i.e. muscle length held constant during the contraction) depends on the length at which it is fixed (Figure 5). It produces large forces at intermediate 20 lengths and relatively smaller forces at either shorter or longer lengths (Scott, Brown et al. 1996). At some intermediate length, typically referred to as optimal length, overlap between actin and myosin filaments is maximal. Therefore, the number of cross- bridges that could form and the contractile force are also maximal. Myofilament overlap is less at longer lengths so the number of potential cross-bridges and force are also smaller. Muscle force decreases at incrementally longer lengths until the point at which no more cross-bridges can form and muscle can no longer produce active force. At relatively shorter lengths, actin filaments start to slide passed each other (i.e. double overlap of actin), which is thought to sterically hinder the formation of cross-bridges, therefore, reducing the amount of force that can be produced. 21 Figure 2-5: Force-Length relationship Muscle force dependence on length is shown for tetanically stimulated muscle with minimal motion of its myofilaments. Overlaid graphics depict myofilament overlap and effects on cross-bridge formation for different lengths. The force-length relationship, as well as all other relationships in the model, is normalized so that it generalizes across a wide range of muscle morphometries. Appropriate normalization factors were obtained through trial and error where the experimental relationships were scaled by various experimental parameters and the parameter producing the most congruent results across all specimens was chosen. Identifying the appropriate normalization factor was sometimes possible through a deeper understanding of the mechanism underlying the phenomenon being modeled. Because the force-length relationship arises from changes in myofilament overlap, it is sensible to assume that a similar force profile would be centered on the optimal muscle length across all specimens; optimal length would therefore be good 22 normalization factor along that dimension. The force corresponding to optimal length is by definition maximal so, clearly, normalizing all force profiles to maximal isometric force makes the relationships more congruent along the other dimension. The equation capturing the normalized relationship is shown below. ( ) − − = ρ β ω 1 exp ce ce L L FL ω, β, ρ are constants that define the exact shape of the inverted-U and are all fiber-type dependent. Force-velocity (FV) Maximally stimulated muscle at a particular length produces different levels of force depending on the rate at which it is shortening or lengthening; it produces less force while shortening and more force while lengthening, relative to the isometric condition (Figure 6; Scott, Brown et al. 1996). Shortening muscle is accompanied by motion of the attached cross-bridges toward relatively less strained configurations. Larger rates of shortening increase the probability that a given cross-bridge will be in a low-strain configuration; therefore, the force generated by the entire population is lower overall. By contrast, cross-bridges in lengthening muscle become relatively more stretched and generate more tension until they are forcibly ripped away from the actin binding sites. 23 Figure 2-6: Force-Velocity relationship Dependence on muscle force output on the velocity of stretch (V>0) and shortening (V<0) for tetanic stimulation at optimal muscle length. The graphics illustrate the hypothesized mechanisms for force enhancement relative to isometric in the lengthening case and depression in the shortening case. Note that in reality myosin heads do not move synchronously as shown in the figure because they are subject to thermodynamic noise and other stochastic events; the graphic illustrates their configuration on average. The equation below is included in the model to account for this phenomenon. ( ) ( ) ( ) [ ] ( ) [ ] ( ) > + + + − ≤ + + − = 0 , 0 , , 2 2 1 0 1 0 max max ce ce v ce ce v ce v v v ce ce ce v v ce ce ce V V b V L a L a a b V V L c c V V V L V FV c v0, c v1, and V max are constants that define the shortening end of the relationship and depend on fiber type. V max corresponds to the maximum velocity of shortening 24 that the muscle fiber can undergo. The lengthening portion of the relationship is defined by fiber type dependent constants b v, a v0, a v1, and a v2. Note that the force- velocity relationship also depends on muscle length. See Potentiation for a detailed description of this phenomenon as well as a mechanistic explanation. Interactions among length, velocity and activation Length dependency of calcium kinetics Changes in length of the sarcomeres has effects on the calcium kinetics that govern activation. The cisterns from which the calcium is released appear to be tethered to the Z-plates at a location that is near the middle of the actin-myosin overlap when the muscle is at optimal length. At longer lengths, calcium has to diffuse over longer distances to get to the actin binding sites, which takes more time. Thus higher firing rates are required to achieve a given amount of activation (Figure 7) and the rise times of the activation are longer (see Brown, Cheng et al. 1999; Brown and Loeb 2000). 25 Figure 2-7: Length Dependence of Cross-bridge Activation Cross-bridge activation is measured as the percentage of maximal force generated for a given amount of myofilament overlap (i.e. effects of myofilament overlap are removed from muscle force). Muscle length is fixed at three different values. In general, a larger portion of the available contractile machinery is activated for the same firing rate when the muscle is stretched. The middle curve corresponds to optimal length. The precise slope of this relationship, i.e. the sensitivity of force to firing rate, is affected by additional factors. A low firing rate activates only a small portion of the binding sites on the actin filaments. An incremental increase in the firing rate results in a relatively larger increase in the number of cross-bridges formed. This is probably due to a relatively larger amount of exposed binding sites per active troponin molecule; this mechanism is known as cooperativity (Gordon, Homsher et al. 2000; Loeb, Brown et al. 2002). Activation of troponin induces a local conformational change in tropomyosin, exposing neighboring binding sites on the 26 actin. When more troponin molecules are activated along tropomyosin, a more global conformational could be occurring, thus, freeing up more actin binding sites. Crossbridge formation may also facilitate exposure of binding sites by inducing relative motion between tropinin and tropomyosin. Potentiation It was originally observed that the force generated by a single twitch of a fast- twitch muscle was much larger after a brief tetanic contraction than before. The effect was called post-tetanic potentiation and the mechanism was unclear. In fact, the potentiated state tends to prevail for minutes after a few, brief trains of stimulation at physiological rates, suggesting that the normal operating state of fast- twitch muscle is the potentiated state and that the conditions observed after a long period of quiescence is a dispotentiated state. Furthermore, the dispotentiated state exhibits rather odd behavior such as a pronounced rightward shift in the shape of the force-length curve at subtetanic frequencies. For these reasons, the model of the fast-twitch muscle fibers presented here captures their behavior in the fully potentiated state. Potentiation and dispotentiation phenomena have shed light on details of muscle mechanics and physiology. Because a muscle fiber does not gain or lose volume as it changes length, the fiber must have a larger diameter when it is at a shorter length. The hexagonally packed lattice of myofilaments in each sarcomere will then be more widely spaced, changing the distance between the myosin heads and the thin filaments where they must bind to form cross-bridges. The heads are located on the 27 ends of hinged arms called myosin light chains, which are canted away from the longitudinal axis of the thick filament by an angle that depends on the phosphorylation of the light chain. During dispotentiation, the light chain becomes dephosphorylated and the cant angle becomes smaller, leaving the myosin heads further from the thin filaments at short lengths than at long lengths. This adversely affects the kinetics of cross-bridge formation in general but more so at low firing rates when exposed binding sites are more sparsely distributed. After a few cycles of calcium release in an active muscle, the short chains become fully phosphorylated and the cant angle increases so as to position the myosin heads more favorably at short and optimal muscle lengths. At longer lengths and narrower lattice spacings, the myosin heads probably remain favorably positioned because they cannot get by the closely spaced thin filaments. The force-length relationship thus returns to the simple inverted-U with maximum at optimal myofilament overlap regardless of firing rate. Slow twitch muscle appears to lack such a dephosphorylation process and so behaves always as if fully potentiated. The function of the dispotentiation remains obscure but may be related to minimizing “stiction”, which arises when stray cross-bridges in an inactive muscle form spontaneously and resist passive stretching by antagonist muscles. The effects of sarcomere length on myofilament lattice spacing give rise to length-dependencies of the force-velocity relationship that are different for slow- and fast-twitch muscle (Figure 8). For slow twitch muscle, the force-velocity relationship depends on length only for the lengthening condition, while fast twitch 28 muscle exhibits a dependency for the shortening condition. In lengthening slow- twitch muscle, smaller spacing between myosin heads and binding sites would result in a higher probability of crossbridge formation, thus, leading to higher forces produced for the same velocity of stretch. This effect is also observed in dispotentiated fast-twitch muscle (see Potentiation), but not in the potentiated state, because the reduction of interfilament spacing resulting from potentiation is thought to mask the effects of length changes (Brown and Loeb 1999). In shortening fast-twitch muscle, it is thought that smaller interfilament spacing at longer lengths increase the probability that a given crossbridge remains attached, thus, making it more likely that it occupies relatively low-strain, low force configurations. Crossbridges in slow-twitch muscle stay attached for longer periods of time, which may mask the effects of length on the rate of detachment hypothesized for fast twitch muscle. 29 Figure 2-8: Length Dependence of Force-Velocity. Force-Velocity curves depicting a substantial length dependency of slow-twitch muscle in the lengthening state (left) and fast-twitch muscle in the shortening state (right). See text for details. Yield (Y) When slow twitch muscle is submaximally stimulated at constant length and then stretched or shortened, its force output declines (Joyce, Rack et al. 1969). This effect intensifies for lower stimulation frequencies, as shown in Figure 9. The mechanism for this phenomenon has been hypothesized to lead to a reduction in the number of crossbridges attached. As muscle length changes past the stroke length of the attached crossbridges, myosin heads detach from their binding sites and must reattach to continue to produce force. The rate at which myosin heads (in slow- twitch fibers) can attach appears be relatively slow compared to the rate at which binding sites slide past them, therefore, making it difficult for crossbridges to reform and produce force. At lower frequencies of stimulation, relatively few binding sites are available on the helical actin, making it less likely that they will be in an 30 appropriate orientation and distance from myosin heads for binding. By applying a time-varying scaling factor to the effective firing rate parameter (see Calcium kinetics and cross-bridge activation), the model is able to capture this effect well (Figure 9). ( ) ( ) Y Y ce Y T t Y V V c t Y − − − − = exp 1 1 & c Y and V Y are constants that define the intensity of the yielding effect and T Y defines its time course. Figure 2-9: Yielding Plot of Force-Velocity relationships of slow-twitch muscle for various frequencies of stimulation. As opposed to tetanic stimulation, the force-velocity relationships for submaximal firing rates are marked by a sharper decline in force in both shortening and lengthening states relative to isometric. Experimental data and model predictions, with and without accounting for yielding, are shown. Figure source: Fig 10c from Brown et al. 1999. 31 Passive elastic elements Muscle is comprised of many elastic components that generate passive forces in response to tensile or compressive strain. The aggregate force of these components may be substantial relative the active force depending on the muscle’s condition of use. Parallel elastic element (F pe1 ) Inactive muscle is under tension for the majority of its anatomical range of lengths. Its passive tensile force rises exponentially over a relatively low range of lengths followed by a steeper linear increase that persists all the way until maximum anatomical length (Figure 10). The precise nature of the relationship for a particular muscle depends mainly on its maximum isometric force and maximum anatomical length. It is negligible over most of the range and peaks at less than 10% of F 0 at maximum anatomical length. 32 Figure 2-10: Parallel Elastic Force. Passive muscle force is plotted over a representative muscle’s range of motion. In general, muscles with larger anatomical ranges of motion produce less passive force at the same length (relative to optimal), that is, their passive force-length curve shifts to the right. As depicted by the superimposed schematics, this force is mainly the result of stretching an elastic element in the sarcomere called connectin that links the thick filament to the Z-disks. Myosin protrusions and actin binding sites are not shown for clarity. In frogs, this passive tension appears to arise primarily from stretching of the spring like connectin myofilaments that tether the myosin molecules to the Z-disks (Magid and Law 1985). The contribution of intermediate filaments like desmin that interconnect myofibrils within a muscle fiber has been shown to be negligible over the physiological range of sarcomere lengths (Wang et al. 1993). Compared to frogs, mammalian muscles have variable but considerably more endomysial collagen surrounding each muscle fiber. This may account for substantial variation in their 33 passive force curves, which can be substantially shifted to the left or right with respect to the optimal sarcomere length defined by myofilament overlap (Brown, Liinamaa et al. 1996). This requires introduction of an additional morphometric parameter into the muscle model, L max, which is based on the maximal anatomical length experienced by the muscle when attached to the skeleton. Below is the equation capturing the effect of this property on the entire passive force-length relationship. ( ) ce r ce ce ce ce pe V k L L L k c V L F η + + − = 1 exp ln , 1 1 max 1 1 1 c 1, k 1, and L r1 are constants defining the precise shape of the curve. Linear damping was also incorporated to the equation with constant coefficient, η, to account for the small amount of damping observed experimentally, which also contributes to the mathematical stability of the model. Thick filament compression (F pe2 ) At very small lengths, muscles generate passive force that opposes the force of contraction. This force is negligible for long lengths but starts to increase exponentially from 0.7L0 until the minimum physiological length (see Figure 11; Brown, Scott et al. 1996). The steady-state force produced by tetanically stimulated muscle decreases at a fairly constant rate for the range of lengths from L 0 and 0.7L 0. At this relatively small length, however, the slope steepens abruptly and remains steep until the 34 length at which net muscle force is zero. Assuming that the effects of double- overlap on cross-bridge formation are the same over the entire range of lengths in which it occurs, the number of cross-bridges, hence force, should decrease proportionally with decreasing length. The abrupt change in force is likely to reflect a collision between the myosin filament and the Z-disk which would generate a reaction force opposing the force of contraction. The extent of compression of the myosin filament onto the Z-disk would be less for submaximal contractions because the number of cross-bridges generating force would be relatively smaller. Thus it is important to model this effect separately from the force-length relationship, which affects force generation equally at all activation levels. The equation below computes the force at the muscle level as a function of length that opposes the force of contraction. ( ) ( ) [ ] { } 1 exp 2 2 2 2 − − = r ce ce pe L L k c L F Constants c 2 , k 2 , and L r2 define the exact shape of the relationship depicted in Figure 11. 35 Figure 2-11: Thick Filament Compression Plot shows the parallel passive force of muscle opposing the contractile force as a function of fascicle length. The schematics illustrate the hypothesis that this force arises due to compression of the thick filament as the Z-disks are pulled really close together at short muscle lengths. Series elastic element (tendon + aponeurosis; F se ) Muscles exert forces on bone segments through intermediate connective tissue, comprised mostly of aponeurosis and tendon. This serially connected tissue affects the muscle’s motion, which modulates force output substantially. Clearly, the length of a muscle for a particular joint configuration depends on the length of the series elastic element because together they must span the entire path from the origin to the insertion site of the musculotendon (see Models of Musculoskeletal Mechanics). The portion of the path that is occupied by muscle depends not only on the resting length of the series elastic element, but also on its elasticity, which determines how much it will stretch when it is pulled by the attached muscle. If a muscle is activated 36 and then deactivated under isometric conditions as defined at the origin and insertion, the sarcomeres may experience large changes in length and especially velocity that will affect their force generating ability. Under dynamic conditions of activation and kinematics, the muscle fibers may actually move out of phase with the musculotendon (Zajac, Zomlefer et al. 1981). This has important implications for the ability of muscle spindles to encode musculotendon length, hence joint position (Hoffer, Caputi et al. 1989; see Models of Proprioceptors). The magnitude of the effects described above depends on the ratio of the lengths of the muscle fascicles to the connective tissue in series with them. Highly pinnate muscles have short muscle fibers oriented obliquely between fascial planes called aponeuroses that merge with the external tendon. Proximally, the aponeurosis has a thin sheet-like structure with a large surface area to accommodate insertion of a large number of fibers while more distally its shape gradually becomes thicker and smaller in surface area. It has been shown experimentally that for a wide range of muscle forces, aponeurosis and tendon strain are the same (Scott and Loeb 1995). The fact that the gradual thickening of the aponeurosis is paralleled by an increase in the number of fibers that exert force on it suggests that the stresses are fairly constant along its entire length. Because the stress and strain experienced by the aponeurosis and tendon are roughly the same then so should their material properties; therefore, mathematically they could be lumped into a single element. With the assumption that tendon force scales with muscle cross-sectional area, or equivalently it scales with F 0, and strain is measured with respect to tendon length 37 at F 0 (L 0 T ), a generic relationship can be derived (see Figure 11). This has been confirmed using several force-length relationships of the tendon/aponeurosis obtained experimentally (Brown, Scott et al. 1996). L 0 T is a good normalization factor because it can be determined more reliably as opposed to the more commonly used tendon slack length. The equation that defines this relationship is shown below. ( ) + − = 1 exp ln T T r se T T se se k L L k c L F Constants c T , k T , and Lr T define the precise form of the relationship shown in Figure 12. 38 Figure 2-12: Series Elastic Tendon/Aponeurosis Force Force generated by elastic tissue that is in series with muscle as a function of its length. The schematic shows that for a given musculotendon length and low muscle activation (bottom), the tendon will be relatively slack and therefore produce little restoring force. At higher activation levels (top), the muscle pulls more on the tendon and causes a larger restoring force. The stiffness is also greater in this case, denoted by the increased slope of the relationship, because the tendon becomes tauter at these higher force levels. Energetics Muscle contraction is an active process whose energy is supplied directly by the high-energy phosphate bonds of ATP molecules (E initial). The expended ATP molecules must then be replaced via metabolism in the muscle to maintain fuel supply for future contractions; this process requires additional energy (E recovery). Researchers measure the rate at which muscles consume energy experimentally by 39 summing the heat output rate and power associated with brief contractions under various conditions. Characterizing the rate of change of energy consumption under each condition independently enables the prediction of total energy loss for situations in which all conditions are changing concurrently. Energy driving the contraction (E initial ) To measure the energy that activates the contractile machinery directly, researchers typically investigate relatively brief contractions that have a shorter duration than the time it takes for metabolic pathways to begin consuming significant amounts of energy. Most of this energy fuels two major physiological processes that underlie contraction: calcium reuptake into the sarcoplasmic reticulum (E a) and cross-bridge cycling (E xb). Energy related to ion pumps (E a ) When a muscle fiber is excited at a particular firing rate, voltage gated channels in the sarcoplasmic reticulum release calcium ions down their concentration gradient into the sarcoplasm that then activate the contractile machinery. Eventually the muscle will relax, which requires all of the calcium to be resequestered to deactivate the cross-bridges and restore the original concentration in the sarcoplasmic reticulum. Transporting calcium ions against their concentration gradient is an active process that is mediated by ATP activated pumps that are distributed all along the sarcoplasmic reticulum’s longitudinal tubules. This energy (E a) can be isolated in thermodynamic experiments by measuring E initial 40 when cross-bridges are not allowed to form. This can be accomplished pharmacologically (using N-benzyl-p-toluenesulphonamide) or by stretching the muscle to the point where myofilaments do not overlap. The energy consumed by this process is mainly a function of firing rate, independent of kinematics. It has been shown experimentally that E a is roughly one half of the energy related to cross- bridge cycling when muscle is fixed at its optimal length (see Figure 13). Although incomplete, evidence suggests that this is true over a large range of firing rates, which means that the E a vs. firing rate relationship ought to have a similar shape to cross-bridge activation vs. firing rate in the isometric case. Thus, the E a-firing rate relationship is modeled using the cross-bridge activation function that is scaled to convert it to units of energy (W). The appropriate scaling factor is defined by constants e 3 and e 4 in the equation below. Given its sigmoidal relationship, this component of energy consumption is most sensitive over an intermediate range of firing rates and least sensitive at either extreme. ( ) 0 , , 3 ) ( 0 4 3 = = = ce ce ce eff ff e a V L L f Af e e f E Energy related to cross-bridges (E xb ) The pulling force generated at each end of a muscle represents the sum of many smaller pulling forces generated by a vast number of constituent contractile elements (sarcomeres and their myofilaments) arranged both in series and in parallel. During a contraction, pulling forces are generated on both ends of the sarcomeres synchronously that are transmitted to the ends of the muscle through 41 connective tissue. If the external load is lower than the contractile force, then the Z- disks will come closer together and the muscle will shorten and will perform mechanical work on the load. The myofilaments will slide past each other and the cocked myosin heads will dissipate their stored mechanical energy as they perform this work. When a cross-bridge reaches the angle at which it exerts no force on the thin filament, the myosin head simultaneously binds a new molecule of ATP, desphosphorylates it to extract its chemical energy, and uses it to detach from the thin filament and recock the head. If the external load is greater than the contractile force, the cross-bridges are stretched until they are forcibly detached, consuming no additional ATP and still in the cocked state, so they can immediately reattach to the next available binding site. The energy consumption related to cross-bridge cycling is thus highly dependent on velocity of sarcomere motion (see Figure 13). It increases rapidly with increasing velocity of shortening (i.e. negative velocity) and it falls rapidly to zero for positive velocity of stretch. Under isometric conditions (zero velocity), cross-bridges continue to cycle at a non-zero rate because of thermodynamic noise. 42 Figure 2-13: Partitioning of Energy Driving Muscle Contraction Total rate of energy consumed by muscle as a function of its velocity as well as its partitioning into E a and E xb components is shown for slow and fast-twitch fibers. In this case, muscles were stimulated maximally at their optimal length. See Tsianos et al. 2012 for a derivation of these relationships. Energy related to cross-bridge cycling also depends on length and firing rate, which together determine the portion of the contractile machinery that is active, i.e. the number of cross-bridges that are cycling. The cross-bridge activation (Af) and force-length (FL) relationships together estimate the percentage of potential cross- bridges that form so they can be used to scale the E xb-velocity relationship obtained at tetanic stimulation and optimal length, i.e. the condition at which all potential cross-brides are formed (Af*FL = 1). The E xb relationship at tetanic stimulation and optimal length (E xb curve in Figure 13) is defined by the piece-wise function in the equation below. The relationship for the shortening condition is approximated well by a rational function while the lengthening condition is approximated by a line. Muscles do not absorb 43 energy (i.e. energy consumption cannot be negative); therefore, E xb is equal to zero when lengthening velocity exceeds a critical value defined as V ce0. Constants e 1, e 2, e 3, and e 4 define the precise form of the relationship. Note that e 3 and e 4 are the same constants used in the equation of E a. ( ) ( ) > ≤ < + + ≤ − − + + = 0 0 2 4 3 4 2 4 3 4 3 4 3 2 2 1 , 0 0 , 3 2 0 , 3 * ) ( , , , , ce ce ce ce ce ce ce ce ce ce ce eff ce ce eff xb V V V V V e e e e e e V e e V e e V e V e L FL V L f Af V L f E where + − = 3 4 2 2 4 4 3 0 3 2 e e e e e e V ce Energy related to metabolism (E rec ) The ATP used for activating the contractile machinery must be recovered to maintain the fuel supply for subsequent contractions. This is accomplished via metabolism within the muscle, which lags the contraction by roughly one second and lasts for about two minutes following relaxation. Energy related to metabolism is quantified by eliciting a muscle contraction over a sufficiently brief period (less than one second) and measuring all of the energy consumed after the muscle relaxes. E recovery is typically expressed as a function of E initial in the literature because the level of metabolic activity naturally depends on the fuel supply that needs to be restored. 44 The E recovery/E initial ratio has been investigated mainly for tetanic contractions, but it is reasonable to assume that it is the same for submaximal contractions because theoretically both E recovery and E initial should be directly proportional to the number of ATP molecules expended, assuming the metabolic pathways and cellular environment is identical. The ratio has been shown to differ substantially between slow and fast-twitch fibers (1.5 versus 1, respectively) undergoing oxidative metabolism. The ratio for glycolitic metabolism has not been investigated rigorously, but theoretical predictions suggest that it is similar. These predictions are based on the energy wasted as heat per ATP molecule during catabolism of glucose. Glycolytic metabolism, however, entails the additional energetic cost of transporting the lactate byproduct to the liver, converting it to pyruvate, and redistributing for subsequent catabolism by oxidative pathways. This cost is likely to be substantial but has not been quantified yet. Because metabolism occurs more slowly than the contraction itself, its dynamics must be modeled in order to account for the amount of energy consumed at any given time as opposed to the total energetic cost of a contraction. Leijendekker and Elzigna (1990) investigated the rate of energy consumed over time by slow and fast- twitch muscle following brief contractions. Assuming that the dynamics of metabolism are linear, the rate of recovery energy consumed over time can be estimated for an arbitrary contraction by convolving the energy driving the contraction (E initial) with the system’s response to an E initial impulse, which can be derived easily from the experiment (see Tsianos, Rustin et al. 2012). Using this 45 approximation, the model does a remarkable job at predicting the dynamics of recovery energy for human muscle contracting in vivo. Balancing accuracy and computational efficiency The feasibility of studying large-scale neuromusculoskeletal systems using models depends largely on their accuracy and computational efficiency. The model presented here is highly accurate, but because it models the dynamics of each motor unit independently, it requires many computations to solve for all of their states at any given time. To reduce the model’s computational load, an algorithm was devised that mathematically lumped all motor units of each fiber type into one. This improved the model’s efficiency approximately 10-fold without having significant effects on its accuracy (Tsianos, Rustin et al. 2012). The lumped model computes an effective firing rate that would reproduce the aggregate force output of a realistic number of motor units with a range of firing rates associated with any given excitation level of the motor pool. The weighting function used to compute the firing rate of the lumped unit is dependent on the drive to the muscle because different levels of drive cause motor units to fire at different levels and may also recruit a different set of motor units (see Recruitment- frequency). The dynamics of force generation also depend on the number of motor units firing and their relative activation as well as each of their activation histories. The lumped unit’s dynamics must therefore be modulated by the level of drive to capture this effect. It turns out that a first-order low pass filter applied to the drive 46 with a time constant that is a function of the change in drive is sufficient. It captures most of the complex dynamics and corresponding force output of the validated muscle model for a wide range of excitation conditions (see Figure 10 in Tsianos, Rustin et al. 2012). This modification to the model improved the computational efficiency substantially because modeling calcium kinetics originally required solving two states per motor unit. A realistic model of muscle having 50 motor units would have 100 states (2*50) while the lumped model has 3 (2+1 related to the additional first order filter applied to the drive signal). Parameters needed to model specific muscles Musculotendons are comprised of the same basic elements, such as sarcomeres for contractile elements and collagen for passive elements, whose structural organization and properties lead to substantially different behavior on the whole musculotendon level. Knowledge of musculotendon morphometry and fiber-type specific properties of sarcomeres is crucial for tailoring the generic relationships that comprise the model to the specific muscle of interest. Optimal fascicle length (L 0 ) As mentioned in “Relevant physiology,” sarcomeres have the roughly the same dimensions and experience the same range of length changes during a contraction. Longer muscle, therefore, has more sarcomeres in series and can undergo larger changes in length. This also means that a given change in muscle length of a longer 47 muscle is accompanied by relatively smaller length changes at the sarcomere level, hence a smaller change in myofilament overlap and force produced. Because the dimensionless Force-length relationship (see Force-length) applies to behavior at the sarcomere level, it must be scaled by the length of the muscle to account for this architectural effect on the behavior of whole muscle. The morphometric parameter used for this purpose is optimal fascicle length, which is the distance between the aponeurosis sites measured along the orientation of the muscle fibers at which muscle produces its maximal force isometrically. Fascicle length is measured instead of muscle belly length because muscle fibers in many muscles are oriented obliquely (see pennation angle) with respect to the long axis of the muscle belly. Muscle mass (m) The amount of force muscle can produce depends on the total number of sarcomeres arranged in parallel. Because sarcomeres have roughly the same diameter, this number is proportional to the cross-sectional area of muscle. Assuming muscle has a cylindrical shape with some known length (L ce) and uniform density (ρ=1.06 g/cm 3 ) with muscle fibers running parallel to the long axis, the cross-sectional area can be estimated by the following equation whose only unknown parameter is mass. ce L m CSA ρ = 48 Because this measure of area is not precise, but is closely related to the level of force muscle can produce physiologically, it is commonly referred to as the physiological cross-sectional area (PCSA). Specific tension, ε, or maximal isometric force produced per unit of cross-sectional area, has been shown to be the same across different fiber types (31.8 N/cm 2 ) and can be used to estimate maximum isometric force (F 0) using the following equation: . * 0 PCSA F ε = Muscle fascicle length in the equation is set to L 0, because ε is defined at that length. The modeled muscle’s F 0 is used to scale the dimensionless relationships to obtain force output in absolute units. Fiber composition Although maximum isometric force per unit area is roughly the same for different muscle fiber types, there are substantial differences in their contractile properties. Fiber types differ in terms of kinematic modulation of force output, rise and fall dynamics of contraction, and especially energy consumption (see Energy related to cross-bridges). The recruitment order of motor units also depends on their fiber type (see Frequency-recruitment). It is therefore important that different fiber types are modeled independently and that the fiber composition of the modeled muscle is known accurately. Determining fiber composition is often challenging because of the many classifications of fiber types and because of the difficultly in measuring the 49 attributes that distinguish them. Fibers differ mainly in terms of their myosin isoforms (namely the myosin light and heavy chains) and the size of their sarcoplasmic reticulum and transverse tubule system that together influence cross- bridge kinetics. There are also large differences in mitochondrial content and blood supply, which affect the metabolic cost of muscle contraction as well as the fiber’s susceptibility to fatigue. Because differences in myosin heavy chain isoform and mitochondrial content have the largest physiological effects they are used most commonly to classify fiber types. Slow-twitch (type 1) fibers have a different distribution of myosin heavy chain isoforms than fast-twitch (type 2) fibers that leads to slower cross-bridge kinetics, as evidenced by the relatively slow rate at which they hydrolyze ATP. Type 1 fibers and some type 2 fibers (namely type 2a) have a high volume fraction of mitochondria, which can be assessed indirectly by measuring the concentrations of enzymes such as succinate dehydroginase, for example, which is normally present in mitochondria. These histochemical approaches require a tissue sample, which could be too invasive to obtain from humans. It is also worth noting that the results may be biased depending on the location in which it is obtained. Fast-twitch fibers for example tend to reside in the outer portion in muscle so if a single sample is obtained from this location the measurement would underestimate the percentage of slow-twitch muscle. There are also noninvasive methods for measuring fiber composition such as magnetic resonance spectroscopy but to date they are relatively less accurate, having poor ability to distinguish between type 2a and type 2b fibers. 50 Characteristic firing rate of muscle fiber type (f 0.5 ) Muscles typically fire at rates ranging from 0.5f 0.5 to 2f 0.5, where f 0.5 is the firing rate at which muscle produces half of its maximal isometric force when its length is held at L 0 (see Optimal fascicle length). f 0.5is an important parameter because when cross-bridge activation relationships of different muscles are normalized by this value, they become congruent and therefore a generic relationship can be obtained. This is useful because only this parameter is needed to construct the entire cross- bridge activation relationship for an arbitrary muscle to a high degree of accuracy. Tendon + aponeurosis length (L 0 T ) The tendon plus aponeurosis is composed of tightly packed strands of collagen that are oriented in parallel with the long axis of the musculotendon. As in muscle, the amount a tendon can stretch depends on its length and the amount of force generates in response to a stretch depends on its cross-sectional area (see Series elastic element). Longer tendons are composed of more collagen molecules arranged in series, each having similar material properties and dimensions, i.e. experiencing the same change in length in response to the same force. Longer tendons can therefore experience larger changes in length and will produce lower forces in response to the same stretch. The generic relationship of the series elastic element reflects the material properties of the constituent collagen fibers and should be scaled by the length of the tendon plus aponeurosis to be modeled. The length should be measured when muscle is exerting maximal isometric force on the 51 tendon (L 0 T ). If only the slack length of the tendon is known, then L 0 T can be estimated as 105% of tendon slack length. Maximum musculotendon path length (L max MT ) As mentioned in “Parallel elastic element,” the amount of passive force generated by muscle at a particular length correlates with its maximum anatomical length. Maximum anatomical length is defined as the maximum length a muscle can have across all anatomical configurations of the joints it crosses. This can be estimated by first measuring maximum anatomical length of the musculotendon and subtracting the length corresponding to the tendon (L 0 T ). Pennation angle Muscle fibers of many muscles are oriented obliquely with respect to the tendon and so not all of the force that they generate is transmitted to the bone segments. Even for relatively large pennation angles, however, this loss in total force transmission in isometric muscle is relatively low so this model does not account for this effect and does not require specification of pennation angle. Modeling the effects of pennation angle would be important for the relatively uncommon scenario in which it changes rapidly by large amounts throughout a contraction, as it would alter the kinematics of the muscle fascicles that have a strong influence on force production. 52 Chapter 3: Mammalian Muscle Model for Predicting Force and Energetics during Physiological Behaviors George A. Tsianos, Cedric Rustin, Gerald E. Loeb © 2012 IEEE. Reprinted with permission from: Tsianos, G. A., Rustin, C., & Loeb, G. E. (2012). Mammalian muscle model for predicting force and energetics during physiological behaviors. IEEE Trans Neural Syst Rehabil Eng, 20(2), 117-133. Preface The model of muscle presented here was published in IEEE TNSRE and was instrumental toward this work because it captures a wide range of physiological behavior at a reasonable computational cost. Accurate predictions of force output over a large range of excitation and kinematic conditions ensure accurate motion of the actuated segments that in turn determine the output of proprioceptors that modulate activity of spinal circuits. The model of metabolic energy consumption enables more realistic cost functions for training neuromusculoskeletal systems, which should theoretically lead to neural strategies that are more physiological. 53 Contributions of the Authors George Tsianos and Gerald Loeb wrote the manuscript, designed the energetics model, and verified the physiological relevance of all of its components. Cedric Rustin assisted with the design of the energetics model. George Tsianos designed and implemented the lumped model of motor unit recruitment. Abstract Muscles convert metabolic energy into mechanical work. A computational model of muscle would ideally compute both effects efficiently for the entire range of muscle activation and kinematic conditions (force and length). We have extended the original Virtual Muscle TM algorithm (Cheng, Brown et al. 2000) to predict energy consumption for both slow- and fast-twitch muscle fiber types, partitioned according to the activation process (E a), cross-bridge cycling (E xb) and ATP/PCr recovery (E recovery). Because the terms of these functions correspond to identifiable physiological processes, their coefficients can be estimated directly from the types of experiments that are usually performed and extrapolated to dynamic conditions of natural motor behaviors. We also implemented a new approach to lumped modeling of the gradually recruited and frequency modulated motor units comprising each fiber type, which greatly reduced computational time. The emergent behavior of the model has significant implications for studies of optimal motor control and development of rehabilitation strategies because its trends were quite different from traditional estimates of energy (e.g. activation, force, stress, 54 work etc.). The model system was scaled to represent three different human experimental paradigms in which muscle heat was measured during voluntary exercise; predicted and observed energy rate agreed well both qualitatively and quantitatively. Index Terms—energetics, modeling, muscle, recruitment Introduction Optimization methods have long been used to find unique solutions for the performance of tasks by “over-complete” musculoskeletal systems (Pedotti, Krishnan et al. 1978; Crowninshield and Brand 1981; Pierrynowski and Morrison 1985). Similarly, optimal control theory can be used to adjust gains in feedback systems (Loeb, Levine et al. 1990; Loeb, Brown et al. 1999; Todorov and Jordan 2002). The physiological soundness of the strategies identified with these tools depends on defining performance criteria that are relevant to the task and the organism. Metabolic energy consumption is a teleologically appealing criterion and there are empirical data supporting it for locomotion and reaching (Franklin, Burdet et al. 2003; Franklin, So et al. 2004), but it is difficult to measure or model at the individual muscle level. Instead, cost functions usually include estimates of muscle recruitment, stress or work that may differ greatly from energy consumption depending on factors such as sarcomere velocity and fiber type. 55 Some cost functions have included models of muscle energetics that are more consistent with metabolic cost (Schutte 1992; Umberger, Gerritsen et al. 2003; Bhargava, Pandy et al. 2004). They are based primarily on the heat output rate and power measurements under various conditions of stimulation that together mirror metabolic energy consumption (Smith, Barclay et al. 2005). The main issues with these models is that they rely almost entirely on enthalpy data, which are sparse, at least partly based on non-mammalian species and do not reflect the true energetic cost under many conditions. Energetic data on submaximally stimulated, lengthening muscle are especially limited for mammals, so modelers often resort to data from non-mammalian species. There is even a discrepancy between enthalpy data presented by Barclay et al., 1993 and Barclay, 1996 for shortening, tetanically stimulated mouse muscle, as discussed by Houdijk et al. (2006). Umberger (2003) developed a model of muscle energetics based on Barclay et al., 1993, which Houdijk et al. found were internally inconsistent with mechanical work. Houdijk et al. found that enthalpy and efficiency estimates from Hatze and Buys’ earlier model (1977) were in qualitative agreement with data from Barclay, 1996, which we have used as part of the thermodynamic basis for our model. Models based on enthalpy experiments estimate the energetics of lengthening muscle poorly. This is because during eccentric contractions substantial mechanical energy is stored through various mechanisms such as tendon, aponeurosis, parallel elastic element and cross bridge compliance (Constable, Barclay et al. 1997; Linari, Woledge et al. 2003; Pinniger, Ranatunga et al. 2006; Trinh and Syme 2007), and 56 perhaps also through changes in cross-bridge biochemical state (Woledge, Barclay et al. 2009). The negative work done on the muscle cannot be stored indefinitely nor can it be converted back into chemical energy. The absorbed energy is not released until after the lengthening phase, however, which creates a large discrepancy between the enthalpy measured and the metabolic energy consumed during activation under lengthening conditions. Many models use this enthalpy data, assuming it is equivalent to metabolic cost and therefore make inaccurate predictions for the energetics. Estimating muscle activation using classical Hill-type models also leads to significant inaccuracies in both energetic cost and force production. These models generally do not capture the dependence of activation on length and fiber specific properties such as sag and yield (Brown, Cheng et al. 1999; Brown and Loeb 2000). These effects are particularly strong at physiological firing rates, as opposed to the tetanic frequencies at which experiments are often conducted. Sag refers to the phenomenon in which the force produced by an isometric muscle in response to a constant frequency of stimulation initially rises to a peak value and then declines slowly (Brown and Loeb 2000). This is thought to occur due to calcium-mediated increase in the rate of calcium reuptake, which reduces calcium ion concentration in the sarcoplasm, hence the number of available binding sites. It occurs primarily in fast-twitch muscle and presumably contributes to speed of relaxation. Given this mechanism, the effects of sag reflect a decrease in the number of cross-bridges and therefore should be included in the energetics model. 57 Yielding is the phenomenon associated with the dependence of the force-velocity relationship on stimulation frequency (Joyce, Rack et al. 1969), or conversely the tendency for a given isometric force to decrease if the active muscle is then stretched or shortened at even moderate velocities. It occurs primarily in slow- twitch fibers where the rate of cross-bridge reattachment appears to be a limiting step (Brown, Cheng et al. 1999). Yielding is more pronounced at submaximal activations, perhaps because the limiting factor in attachment rate is related to the stereochemical alignment between the myosin head and exposed cross-bridge binding sites on the helical actin. When all binding sites are available at full activation, a cross-bridge that detaches due to motion is more likely to reattach rapidly and the force will be maintained. At reduced activation, more of the available binding sites will be suboptimally oriented. If slow-twitch myosin is less mobile or slower to attach than fast-twitch myosin, then rate of reattachment might become the limiting step in cross-bridge cycling and, hence, energy consumption. The correlation of yield to muscle stiffness (Malamud, Godt et al. 1996) provides further support for a mechanism related to the number of cross-bridges formed. The validity of models that are based on empirical data depends on the accuracy and richness of the dataset used, neither of which is really satisfactory for muscle energetics. We have used most of the available enthalpy data from mammalian muscle in a model whose structure is closely related to the individual physiological processes underlying activation, force production and energy consumption. This improves the likelihood that the model can be interpolated and extrapolated to 58 predict energy consumption for conditions that have not been studied, and it provides a straightforward mechanism to test its validity and to improve its parameter estimation as new data become available. Methods Model Design and Rationale The energetics of muscle contraction can be separated into an initial component (E initial) that is consumed in phase with the contraction and a recovery component (E recovery) that lags the contraction slightly and lasts up to a few minutes following its termination (Leijendekker and Elzinga 1990). E initial corresponds to the ATP consumed at the time of the contraction and its rapid replenishment through the creatine kinase reaction (Smith, Barclay et al. 2005). E recovery corresponds mainly to the energy consumed by metabolic pathways for restoring ATP and phosphocreatine (PCr) concentration in the sarcoplasm. The energy supplied from ATP molecules fuels two main processes during the contraction: regulating concentration gradients of ions that excite the muscle fiber and its contractile machinery (E a) and cycling of cross-bridges that generate contractile force (E xb). Partitioning energetics among these physiological processes in the model is important for computing energy consumption for untested conditions because it allows us to extrapolate their contributions independently according to the specific stimulation parameters that affect them. 59 Energy Consumption Related to Cross-bridge Cycling The majority of the energy supplied by ATP is consumed due to cross-bridge cycling and is referred to as cross-bridge energy (E xb). For a cross-bridge to form, myosin heads must be in a high-energy, strained configuration containing bound ADP. Such myosin heads bind rapidly to nearby actin sites that are exposed by calcium activation. The elastic energy stored in the myosin head is subsequently released depending on the external load on the cross-bridge. If the load is less than the restoring force of the cross-bridge, then the myosin head will move “concentrically”, reducing its contribution to contractile force toward zero at its equilibrium position, where it has the highest probability of detachment from the actin site. The detachment process is coupled to recocking of the head in the strained configuration, which requires a fresh ATP molecule to displace the ADP. If the load is greater than the restoring force of the attached cross-bridges, the muscle fiber will lengthen, pulling the cross-bridges to even greater strain angles where forces increase but then drop as the cross-bridges are forcibly detached. This “eccentric” detachment leaves the myosin heads in their cocked state, so they can immediately reattach to an available binding site without consuming energy (although the actual rate of reattachment may depend on the relative availability of binding sites, which in turn depends on the level of activation; see description of yielding phenomenon in the Introduction). The amount of energy consumed by muscle due to cross-bridge cycling therefore depends on how many cross-bridges are attached and the rate at which they are cycling concentrically, as opposed to the 60 amount of force that they are generating. The number of cross-bridges formed depends primarily on firing rate, myofilament overlap, and the effects of sag and yield (see Introduction). Cycling rate depends mainly on the velocity of contraction as measured at the muscle fiber, but it does not fall to zero for so-called isometric contractions. This could arise from substantial “squirming motion” of the myofilaments due to their compliance, the pulsatile excitation of muscle fibers, the stochastic distribution of cross-bridges at any instant, and/or thermal vibration of all of the molecular structures. Energy consumption related to excitation The energy associated with ionic excitation is typically referred to as activation energy (E a) and is related to the number of ATP molecules consumed by membrane- bound pumps for sodium, potassium and calcium ions. When an excitatory impulse from a motoneuron is transmitted through the neuromuscular junction and spreads along the muscle fiber and into the transverse tubules, it depends on sodium and potassium ions leaking across the cell membrane down electrochemical gradients. These electrochemical gradients must then be maintained through the Na+-K+ pump at the expense of ATP to allow transmission of subsequent impulses. Action potentials at the points of contact between the transverse tubules and cisternae of the sarcoplasmic reticulum (SR) lead to release of calcium ions from the cisternae down their concentration gradient into the sarcoplasm, where they bind to troponin and induce a conformation change along the thin filament that exposes binding sites 61 for cross-bridges. To regulate calcium flux within the sarcoplasm and restore the calcium concentration gradient, additional ATP is expended to pump calcium back into the SR, allowing the muscle to relax and restoring calcium to the cisternae. The component of E a that is related to action potentials along the sarcolemma is small and generally proportional to the firing rate of the motor unit. The component of E a that is related to calcium activation of cross-bridge binding is larger and initially proportional to firing rate but tends to plateau at higher frequencies because of temporary depletion of calcium in the cisternae (Somlyo, Gonzalez-Serratos et al. 1981). Recovery Energy The recovery energy is the energy required to restore the ATP and PCr concentrations in the sarcoplasm. ATP can be synthesized via aerobic or anaerobic metabolic pathways, after which it may react with creatine to form PCr. Muscle undergoes anaerobic metabolism when oxygen levels are insufficient for oxidative pathways, which tends to occur during sustained, strong contractions of muscles that generate hydrostatic pressures sufficient to occlude capillary blood flow. These metabolic pathways have different efficiencies, meaning that they consume different amounts energy per ATP or PCr molecule synthesized. The total energy consumed in this phase, therefore, depends on the amount of ATP and PCr needed to be restored and on the specific metabolic pathways involved. Fast-twitch muscle fibers (type IIb) have a low oxidative capacity but large amounts of stored glycogen, which 62 they can metabolize anaerobically, albeit inefficiently, to replenish ATP during strong contractions. Slow-twitch (and type IIa fast-twitch fibers) muscles have little stored glycogen but they have a high oxidative capacity to restore ATP efficiently from the complete oxidation of glucose absorbed from blood flow. Implementation of the Model In thermodynamic experiments, the sum of heat and work measured during isovelocity contractions and isometric contractions of brief duration reflects E initial. The heat that is produced largely after the contraction is referred to as recovery heat and corresponds to E recovery (see (Wendt and Gibbs 1976; Crow and Kushmerick 1982; Leijendekker and Elzinga 1990; Barclay, Arnold et al. 1995)). The model for E initial is based on thermodynamic data characterizing the relationship of energy to shortening velocity (Barclay 1996) for both slow- and fast- twitch muscle and division into contributions from E xb and E a (Barclay, Woledge et al. 2007; Barclay, Lichtwark et al. 2008). The data, however, are obtained mostly from tetanically stimulated muscle at optimal sarcomere length. Virtual Muscle (VM) incorporates models of processes such as gradual recruitment and frequency modulation of different muscle fiber types and their interaction with force-length and force-velocity relationships for those fiber types. Thus the addition of an energetics model required estimation of coefficients to rescale existing terms for excitation (firing rates) and force generation (newtons, N) into units of energy consumption (watts, W). We also redesigned a continuous recruitment algorithm 63 for VM based on the lumped motor unit scheme presented in (Song, Raphael et al. 2008), so that it is both computationally efficient and able to provide accurate estimates of both muscle energetics and force in muscles with various mixtures of two different fiber types (see Appendix A). The model and its implementation depend on certain assumptions: 1. All mammalian skeletal muscles use similar chemical pathways and molecular mechanisms to convert metabolic energy into mechanical work, thereby allowing us to pool data from different species. 2. The energy cost of all physiological processes is a function of ATP regenerated to restore the initial conditions, which is not related strictly to the chemical or mechanical work performed or heat liberated when that work was performed. 3. Processes related to ATP regeneration occur mostly within the contracting muscle rather than in other organs (e.g. liver) that are involved in metabolism, therefore, enthalpy measurements from isolated muscle preparations are good approximations of energy cost. 4. The metabolic cost of ATP regeneration via a given chemical pathway is independent of operating temperature (Smith, Barclay et al. 2005), thereby allowing us to utilize data from preparations typically below normal body temperature to preserve their viability. The actual energy required to drive a given chemical reaction (e.g. regeneration of ATP from PCr) is a function of temperature, but the energy available in the biological source (PCr) is higher 64 than that by a safety margin. As long as the reaction takes place at all, the total energy consumed will be constant and equal to the sum of chemical energy stored in any products (e.g. ATP) plus excess energy dissipated as heat and/or work. The energetics algorithm (refer to the block diagram in Fig. 1) is divided into sections that correspond to the major physiological processes underlying energy consumption. Equations are provided in Table 1 and symbols are defined in Table 2 of Appendix A. The sequence and specific form of the computations was determined by the experimental data that were available. E a, for example, has been investigated relative to E xb rather than as an explicit function of firing rate, so in order to calculate its contribution to total energy expenditure for submaximal stimulation, we computed E xb in absolute units first. Similarly, E recovery is defined in the literature as a multiple of E initial so we had to compute E initial first to obtain E recovery. Energy Consumption Related to Cross-bridge Cycling To extract the contribution of cross-bridge cycling from the E initial-velocity relationships presented in (Barclay 1996) for tetanically stimulated muscle at optimal sarcomere length, we calculated the contribution of E a based on the finding from (Barclay, Woledge et al. 2007; Barclay, Lichtwark et al. 2008) that it is approximately one third (parameter “a” in the model) of the initial energy consumed in the isometric condition. Note: E initial differs significantly for each fiber type and was therefore handled separately in the model. 65 E f = f = a ∗ E f = f , L = L , V = 0 (Fig. 1, STEP 1) Because E a is assumed independent of contractile velocity, we removed this value from the entire E initial-velocity relationship to obtain the shortening E xb- velocity relationship for tetanically stimulated muscle at optimal sarcomere length. E f = f , L = L , V ≤ 0 = E f = f , L = L , V − E f = f (Fig. 1, STEP 2) The E xb-velocity relationship for lengthening muscle was designed to capture the observation that ATP consumption rate in actively lengthening muscle decreases to a low steady level near zero at a modest lengthening velocity (see (Butler, Curtin et al. 1972; Curtin and Davies 1973)). Because the energy over this range of velocities is relatively small and constant, it can be assumed that it corresponds almost entirely to E a (i.e. E xb =0). Thus, we computed the energy rate due to cross-bridge cycling by linearly extrapolating the shortening E xb-velocity relationship from the isometric condition down to zero energy. E xb was set to zero for larger lengthening velocities. Note that the linear approximation has negligible effects on the model’s accuracy for most behavioral tasks, as E xb during lengthening has a relatively small contribution to the overall energetic cost. E f = f , L = L , V > 0 = E f = f , L = L , V = 0 + d dV [E f = f , L = L , V]| ! ∗ V" ∗ ifE ≥ 0 66 (Fig. 1, STEP 3) To obtain E xb for physiological firing rates and kinematics, we scaled this relationship with the Acteff (=rFpcsa*W*Af), and FL terms in VM, which provide a measure of the number of cross-bridges formed depending on these conditions. Note: Acteff and FL are different for each fiber type due to type-specific biochemical and mechanical properties of the contractile machinery. The sag phenomenon in fast-twitch fibers and yield in slow-twitch fibers, for example, are incorporated into the model through the Af term of VM (see (Brown and Loeb 2000) and (Joyce, Rack et al. 1969) for more details on modeling of sag and yield, respectively). The role of these terms in VM’s energetics and force models is shown schematically in Fig. 10 (see Appendix A for an overview of VM). E f, L , V = Act (( f, L ∗ FLL ∗ E f = f , L = L , V (Fig. 1, STEP 4) Energy Consumption Related to Excitation Researchers typically study E a relative to E initial or E xb rather than as an explicit function of firing rate, so we scaled E xb to calculate its contribution to total energy expenditure. To measure E a in intact muscle, experimentalists record the heat output of muscle stimulated isometrically under conditions that prevent cross- bridge formation. This can be done mechanically by stretching the muscle so as to eliminate myolfilament overlap or pharmacologically by exposure to BTS (N-benzyl- p-toluenesulphonamide) to inhibit the binding of myosin heads to actin sites 67 (Barclay, Woledge et al. 2007; Barclay, Lichtwark et al. 2008). Both techniques produce similar results for slow- and fast-twitch mammalian muscle at temperatures within the range of 20 to 30 degrees Celsius. It has been shown that the energy expended by excitation is roughly one third of E initial consumed by muscle that is contracting isometrically at optimal sarcomere length. Equivalently, the activation energy consumed is one half of the energy consumed due to cross-bridge cycling (given that E initial = E xb + E a and a=1/3). E f = a 1 − a ∗ E f, L = L , V = 0 (Fig. 1, STEP 5) Recovery Energy The enthalpy resulting from ATP and PCr resynthesis is naturally related to the ATP and PCr expended, which are the major contributors to the initial enthalpy measured. E recovery is therefore determined based on the initial heat rate and efficiency of the metabolic process that restores these molecules. The ratio of recovery to initial enthalpy (parameter “R” in the model) under aerobic conditions has been measured to be 1 for extensor digitorum longus (EDL; predominantly fast- twitch) and 1.5 for soleus (SOL; predominantly slow-twitch) muscles (Leijendekker and Elzinga 1990). The recovery to initial enthalpy ratio has not been investigated rigorously for the anaerobic condition but it is theoretically close to that of the aerobic condition (see (Leijendekker and Elzinga 1990)); therefore, we chose to model the efficiency of both forms of metabolism identically. Although the 68 dynamics of recovery are relatively slow, lasting on the order of a few minutes (Crow and Kushmerick 1982), and are impractical to study in active muscle (Smith, Barclay et al. 2005), it seems sensible for the purposes of this model to represent E recovery as a multiple, R, of the initial heat rate. Note that metabolic processes, hence R, are different for each fiber type. E +,-+. f, L , V = E f, L , V ∗ R, where E f, L , V = E f, L , V + E f (Fig. 1, STEP 6) This formulation does not model the instantaneous recovery rate accurately, but it provides a good approximation of the total recovery cost of a particular motor task. Also note that R as well as other components of the model may vary with muscle fatigue (Barclay, Arnold et al. 1995; Barclay 1996). Further experiments are needed to understand the process of fatigue before its effects on energetics and force can be modeled accurately (see Discussion). Total Energy The total energy rate was taken as the sum of the E initial and E recovery. A conversion factor (M) obtained from (Kelso, Hodgson et al. 1987) was then used to convert the energetic units from mW/gdry to mW/gwet because the wet mass is a parameter specified in VM and is more readily obtainable from the literature. E , f, L, V = 4E f, L, V + E +,-+. f, L, V5 ∗ M, in mW g ; (Fig. 1, STEP 7) 69 Figure 3-1: Overview of the energetics algorithm. Thermodynamic data for tetanically stimulated muscle at optimal sarcomere length provide the basis for the energetics model. Data from force production experiments that characterize underlying physiological processes were used to generalize the model across a wide range of firing rates and sarcomere kinematics. Gray elements in the diagram are dependent on fiber type. See text for details. 70 Overall Behavior of the Model In this section, we verify the implementation of the model and demonstrate its emergent behavior across a wide range of physiological conditions. We conclude the section with a comparison of energy and force output to show their qualitative differences and emphasize their different implications when used as teaching signals for an optimization algorithm. Model verification The main experimental relationships on which the model is based are reproduced accurately by the model. Fig. 2 shows the E initial-velocity relationship obtained experimentally (open squares) and that predicted by the model (solid trace) for tetanically stimulated muscle at optimal sarcomere length. The experimental and predicted relationships for both 100% slow- and 100% fast- twitch muscle agree very closely. The model’s prediction is also within the experimental range of E a (30-40% of E initial) obtained from (Barclay, Woledge et al. 2007; Barclay, Lichtwark et al. 2008). Although there are some published E a values that are significantly higher than this range, we chose not to include them as their underlying experimental techniques are questionable (Barclay, Woledge et al. 2007). E a, as computed by the model, is independent of velocity because it is assumed that myofilament motion does not affect active ion transport. 71 Figure 3-2: Model verification The model accurately reproduces the energy-velocity relationships presented in (Barclay 1996) and excitation energy as a percentage of total energy consumed presented in (Barclay, Woledge et al. 2007; Barclay, Lichtwark et al. 2008), (Barclay, Woledge et al. 2007; Barclay, Lichtwark et al. 2008). Effects of Fiber Composition, Excitatory Drive and Sarcomere Kinematics on Energy Consumption Energy consumption and its partitioning have qualitative dependencies on fiber composition, excitatory drive, and sarcomere kinematics. As shown in Fig. 3, fast- twitch fibers can consume more than five times as much energy per unit time as slow-twitch fibers for the same force produced (both muscles were contracted isometrically at optimal sarcomere length). In both slow- and fast-twitch muscle, energy consumption rises with neural drive because it increases the rate of ionic transport, hence E a (as shown by the larger red portion of the column in Fig. 3). A rise in drive also increases the number of cross-bridges formed, leading to an increase in E xb (as shown by the larger blue portion in Fig. 3). E recovery (marked green in Fig. 3) increases as well because it is proportional to the sum of E a and E xb. 72 Energy consumption also has a strong dependence on sarcomere velocity, as it increases significantly with decreasing velocity (i.e. higher rates of shortening). As velocity decreases, E xb rises because cross-bridges complete their cycle at a higher rate. E a remains the same because energy related to ionic transport is independent of velocity. Energy consumption is reduced when sarcomere length is either shortened or stretched beyond L0. This is because in both cases there is a reduction in the number of cross-bridges formed, which reduces E xb. The asymmetry mirrors the force-length relationship and is related to the different mechanisms underlying these two cases: reduced myofilament overlap in the stretched sarcomere and double myofilament overlap in the shortened sarcomere. E a remains the same because it is independent of sarcomere length. 73 Figure 3-3: Overall behavior of the model 74 This figure illustrates qualitative dependencies of energy consumption on fiber composition and various stimulation parameters. Stimulation parameters are shown on the right, with the varied parameter labeled in bold (U-excitatory drive, V-sarcomere velocity, L-sarcomere length). The bars within the shaded region represent energy consumption for an identical set of stimulation parameters. All bars are drawn to scale to emphasize the relative effects on energy consumption resulting from different stimulation conditions. Energy is partitioned according to the major physiological processes that underlie it to illustrate the dependency of each process on the varied parameter. Comparison of Energy and Force Output We computed initial energy consumption and force production for a model of human biceps longus (40% S, 60% F) as a function of fascicle velocity from -6 to +3 rest lengths per second (L0/s). These two results are approximately inversely related for a family of curves for neural drive in 20% steps from 0 to maximal excitation (see Fig. 4), where fiber types are recruited and frequency modulated according to the size principle (Cheng, Brown et al. 2000). Maximal force of ~840N (140% of maximal isometric force) occurs for stretching velocities greater than 1L0/s when energy consumption is ~40W. Maximal energy consumption of ~330W occurs near Vmax (~-6L0/s), when force drops to zero. Energy consumption per unit isometric force rises from 0.1W/N at 40% drive (all slow-twitch fibers) to 0.2W/N at 100% drive. 75 Figure 3-4: Comparison of force production and energy consumption Comparison of force production (top) and total energy rate (middle) under identical conditions of stimulation. Energy and force-velocity relationships are shown for a representative muscle analogous to the long head of the biceps (40% slow/60% fast-twitch fibers; mass = 300g; F0 = 600N; L0 = 16cm). Velocities ranged between -6L 0/s and 3L 0/s and excitatory drive ranged from 20% to 100% as shown next to the plots on the right. Lower graph shows initial energy rate (partitioned into E xb and E a) for ±1 L 0/s and midrange excitatory drives (U = 20%, 40% and 60%MVC), which reflect common conditions of use. 76 It is evident from Fig. 4 that force and energy measures are vastly different and they will probably lead to different neural strategies when they are incorporated into an optimization algorithm’s cost function. A cost function based on muscle force (or related parameters such as torque, stress etc.), for example, would lead to preferential recruitment of muscles that are shortening, whereas a cost function based on energy would lead to recruitment of muscles that are lengthening. This is particularly relevant for tasks like locomotion where subjects tend to recruit muscles during their lengthening phase to improve energetic economy of the movement (see Discussion). Validation To assess the model’s ability to predict energy consumption of human muscle, we validated it against data from three experiments performed in vivo that were independent from the datasets used to construct the model. We focused specifically on the model’s ability to capture the substantial effects of fiber composition, excitatory drive, and kinematics, respectively. Effects of Fiber Composition To test the validity of the model in predicting the effects of fiber composition on energy consumption, we compared its output to the results from (Bolstad and Ersland 1978). In this experiment, the temperature rate was measured at MVC for a variety of human muscles (soleus, sacrospinalis, biceps brachii) that covered a wide 77 range of fiber compositions (~0-70% fast-twitch fibers). To make a fair comparison between the model and experimental data, we considered that although the subjects were instructed to contract their muscles maximally, they tend not to reach true MVC without intensive training (Allen, Gandevia et al. 1995). (Bolstad and Ersland 1978) did not mention such training nor did it employ the twitch interpolation technique of (Allen, Gandevia et al. 1995) to confirm MVC. The increase in force generated by the stimulation would have indicated how close to maximal the muscle was actually firing, helping identify the true MVC value. Experimental results and model predictions for two cases are shown in Fig. 5. The predicted energy rate is shown for the cases where the subjects attained true MVC (experimental MVC corresponds to U=1) and an underestimated MVC (experimental MVC corresponds to U=0.8). Both cases reproduce the strong positive correlation between percentage of fast-twitch fibers and energy consumption, with the second case (U=0.8) being in close agreement with the data in terms of absolute value and slope. 78 Figure 3-5: Validation results for effects of fiber composition Experimental results and model predictions are plotted for the rate of energy consumption as a function of fiber composition. Muscles were assumed to be contracted isometrically at their optimal length. Fiber composition is labeled on the abscissa as the percentage of fast-twitch fibers present in the muscle. The two traces are shown as model predictions that correspond to scenarios in which subjects achieved maximal drive (MVC @ U=1) and undershot it by 20% drive (MVC @ U=0.8), respectively. Effects of Excitatory Drive To assess the model’s performance for submaximal contractions, we compared its predictions to results from Saugen and Vollestad (Saugen and Vollestad 1995). Temperature rate of the vastus lateralis muscle was measured during isometric contractions for forces ranging from 10 to 90% MVC. Assuming that the thermocouple measurements represented the average temperature of the muscle and a negligible amount of heat escaped the muscle, the rate of heat production would be related simply to temperature rate through the heat capacity of muscle. Thus, to compare our model’s predictions to the temperature measurements, we converted energy consumption (which is equivalent to heat liberation in this case) 79 to temperature rate by dividing it by the heat capacity of muscle (3.75 kJ/kg; (Saugen and Vollestad 1995)). As in the previous section, we considered that the subjects in this experiment may have not reached true MVC. Experimental results and model predictions for the cases where true MVC was attained or underestimated are shown in Fig. 6. If we make the reasonable assumption that subjects undershot the true maximal drive by 20% we get a substantially better agreement in terms of the absolute value and qualitative relationship between temperature rate and excitatory drive. Nevertheless, there is some deviation between the model output and experimental results. The model probably underestimates energy consumption at low force levels because it does not fully take into account the squirming motion (see Model Design and Rationale) of the myofilaments, which may be more pronounced at these levels because of the nonlinear compliance of biological connective tissue at low stresses. As force increases the squirming motion reduces and the deviation expectedly decreases. Furthermore, their measured heat values include some recovery energy because of the duration of the contractions, which were much longer for the low contractile force (15s for 10%MVC to 6s for 90%MVC). This would also over- estimate initial energy consumption at low forces more than at high forces, flattening the experimental relationship. The plateau of the experimental data above 70%MVC may relate to increasing heat loss into adjacent, cooler tissues. 80 Figure 3-6: Validation results for effects of excitatory drive Experimental results and model predictions are plotted for muscle temperature rate as a function of isometric force. Muscles were assumed to be contracting isometrically at their optimal length. Force on the abscissa is normalized by the force produced at maximal effort. Two traces are shown for model predictions that assume the human subjects truly achieved maximal drive (MVC @ U=1) and undershot it by 20% (MVC @ U=0.8). It was assumed that the vastus lateralis muscle in this study was composed of approximately equal portions of slow and fast-twitch fibers. Effects of Kinematics In order to validate the effects of kinematics on energy consumption, we compared our model’s prediction to observations from (González-Alonso, Quistorff et al. 2000). Their experiment followed the dynamic knee extension paradigm (Andersen, Adams et al. 1985) in which subjects are hooked up to a cycle ergometer and trained to extend their knee from a neutral sitting posture (~80 degrees extension) to nearly full extension and passively return to the neutral posture. The task is designed to recruit only the quadriceps muscles, which are active mostly in the first phase of the extension when the shank and foot are accelerating. Subjects in this study were asked to perform the task with maximal effort (i.e. as fast as 81 possible) for three minutes while the rate of metabolic energy consumption (heat production plus mechanical power output) was estimated. Initially, we anticipated that actual rate of energy consumption would differ significantly from our model’s prediction. Energy rate in the experiment increased by approximately 40% from the beginning to the end of each session whereas our model’s prediction would stay constant because it is solely dependent on muscle recruitment level and kinematics, which are virtually the same across extension cycles. We hypothesized that the observed increase in energy rate was due to the dynamics of the ATP/PCr recovery process, which has a relatively long time constant. While the energy consumed by the recovery process is captured by the model presented here, its dynamics are not (see “Recovery energy” section). To estimate these dynamics we used the energy rate response during the recovery phase of a brief contraction (Leijendekker and Elzinga 1990) as the impulse response characterizing the ATP/PCr recovery system. See Appendix B for details on how the task and associated energy consumption were modeled. As shown in Fig. 7, the model’s prediction matches experimental results very closely, given the reasonable assumption that subjects’ effort was not truly maximal (see previous two sections). The model captures the dynamics of energy consumption and the total amount consumed (model: 35,000J; experiment: 30,000- 40,000J; integral of curves in Fig. 7) remarkably well. For the brief contractions studied by (Leijendekker and Elzinga 1990), the recovery process produces little or no lactate, hence relies on oxidative metabolism, which takes longer to complete and 82 actually generates more heat per unit ATP regenerated. Thus, the time constant and scaling factor for fast-twitch fibers derived from this experiment applies to the first couple of cycles of the cycle ergometry, but quickly shifts to a longer time constant and lower scaling factor as the recovery process shifts toward anaerobic metabolism. The model results in Fig. 7 do not incorporate such a shift, whose dynamics are not known, but either change would tend to improve the agreement between experiment and model. Figure 3-7: Validation results for effects of kinematics Experimental results and model predictions are shown for the dynamic knee extension task (see text for details) performed over three minutes. Both E initial and E recovery contribute significantly to the total energy rate due to the long duration of the task. As in the experiment, the energy rate computed by the model was averaged over one second intervals (equivalent to one cycle of the movement). E initial is constant throughout the entire duration of the task because the stimulation parameters are constant but E recovery increases over time due to the dynamics of metabolism (see Appendix B for details). 83 Discussion Limitations The lumped model described here is currently limited to two fibers types, which we have identified as fast- and slow-twitch; a more general version of the Virtual Muscle modeling software does support arbitrary numbers of motor units and fiber types (Cheng, Brown et al. 2000). In reality, many muscles are composed of a range of fiber types whose force-generating and metabolic properties reflect different mixtures of those that we have associated with our fast- and slow-twitch models (Westerblad, Bruton et al. 2010). Fast-twitch fibers are often divided into type IIb (type IIx in humans), which typically utilize glycolytic metabolism, and type IIa, which typically utilize oxidative metabolism, similarly to slow-twitch fibers. The energetics model is based primarily on experiments performed on mouse soleus (SOL) and extensor digitorum longus (EDL) muscles, which are composed mostly of type I and type IIa fibers, respectively (Luff and Atwood 1971) . In a muscle with a large proportion of type IIb/x fibers, our estimate of recovery energy will likely be too high, as glyocolytic pathways are known to produce less heat per ATP regenerated; the theoretical ratio of recovery to initial energy for glycolytic metabolism is 0.8 whereas that of oxidative metabolism is 1.13 (Woledge, Curtin et al. 1985; Leijendekker and Elzinga 1990). It is important to note that these estimates depend on the energy source (e.g. carbohydrate vs. fat) and other conditions in the sarcoplasm (e.g. ionic and metabolite concentration). Furthermore, 84 the theoretical estimate for glycolysis does not include the energy cost associated with the extra step of handling the lactate byproduct (e.g. converting it to pyruvate for later use in oxidative pathways). More rigorous experimentation and analysis of metabolic efficiency are necessary to assess their actual contribution to muscle energetics. The model should be used with care when simulating tasks in which the muscles involved may experience fatigue because the effects of fatigue on force production and energy consumption are not included in our model. Fatigue, for example, has been associated with significant changes to the energy and force-velocity relationships (Barclay 1996). We chose not to incorporate these effects into the model because they are specific to a particular state of fatigue, which may consist of a mixture of several mechanisms that are too poorly understood to model individually. The ratio of recovery to initial heat also varies with the degree of fatigue in fast-twitch EDL muscle (Barclay, Arnold et al. 1995), which is also not included in the model. The ratio of activation energy to total energy, constant term “a” in our model, may have some dependence on firing rate. There are numerous competing effects on “a” from phenomena that are dependent on firing rate such as calcium release efficiency, mechanical squirm (related to the noise in force output and muscle stiffness) and cross-bridge cooperativity. Unfused contractions elicited by submaximal firing rates, for example, cause larger oscillations in output force and sarcomere length than in the tetanic condition. These oscillations should produce a 85 higher cross-bridge cycling rate than would be expected if the contraction were completely fused as in the tetanic condition and therefore may reduce “a”. The dependence of “a” on firing rate has not yet been investigated systematically but it is likely that it is a small effect based on the finding that the ratio was nearly the same for a submaximal and tetanic frequency of stimulation in fast-twitch muscle (Barclay, Lichtwark et al. 2008). This is somewhat counter-intuitive, because E a might be expected to continue to rise with firing rate even as output force plateaus. It may be that the calcium reservoir in the cisternae starts to deplete at approximately the same time as the sarcoplasmic calcium concentration reaches saturation for troponin binding, reflecting efficient design. Nevertheless, if a salient dependency emerges from new data, the model is sufficiently modular so that the “a” term can be readily modified to account for it. “Motor noise” (fluctuating recruitment of individual motoneurons; (Jones, Hamilton et al. 2002)) produces force fluctuations that act against the series- elasticity of tendons and aponeuroses to produce fluctuations in fascicle and sarcomere length that should increase cross-bridge cycling for a given nominal velocity of the muscle-tendon unit. VM includes an explicit model of the series- elasticity and the lumped recruitment model is designed to generate force fluctuations reflecting whatever model of motor noise is added into the driving excitation signal (Song, Raphael et al. 2008). Therefore, energy consumption associated with motor noise should be correctly incorporated into the predictions of this new version of VM. 86 Role in Motor Control Research Even with an accurate energetics model at hand, using it to gain insight into the neural strategies underlying motor tasks remains a challenge. Although the nervous system appears to consider metabolic energy consumption when selecting movement strategies, it does not always pick the most energy efficient strategies. For example, energetics are obviously less important if the task is to reach to a target 10 times vs. 1000 times. Furthermore, if subjects expect a perturbation while performing a task, they are likely to cocontract more during the movement to resist it, which could be mediated through a reduced emphasis on energetics. There is also evidence that energetic considerations change as subjects learn to perform a task, starting with excessive cocontraction initially and gradually reducing it as they learn to use more efficient feedforward and feedback strategies to improve their kinematic performance (Franklin, Burdet et al. 2003; Franklin, So et al. 2004). Tsianos et al. (2011) studied motor learning in a planar arm model, which was much more reliable when the system started with cocontraction and then learned to perform well without it. Such high level strategizing can be expected to emerge from motor learning when the cost function includes realistic changes over time in the relative importance of kinematics and energetics. The fact that force output and energy consumption actually move in opposite directions around zero velocity may shed light on speculations about the role of biarticular muscles (Van Ingen Schenau 1989; Gielen, Schenau et al. 1990; Lieber 1990; Pandy and Zajac 1991; Kautz, Hull et al. 1994; Kumamoto, Oshima et al. 1994; 87 Neptune and Van Den Bogert 1998; Schouten, de Vlugt et al. 2001). Biarticular muscles tend to be active when the two joints that they cross are moving in opposite directions. The net muscle velocity tends to be close to zero or to reverse sign while these muscles are active, making them specifically well-suited for economical transfer of momentum from segment to segment, avoiding the need to dissipate excessive kinetic energy as heat. The optimization techniques described in the Introduction all require minimization of a cost function to predict the roles of individual muscles for a given task. When such cost functions are applied to engineering problems, they usually consist of a combination of terms related to speed, accuracy and energetic cost. The model provided here permits the development of analogous cost functions for biological systems rather than the arbitrary minimization of total force or recruitment that is generally used instead. Role in Studying Effort and Fatigue VM with energetics can also be used as a tool to understand and model fatigue. Fatigue is a collection of incompletely understood phenomena in which motor units change their properties as a result of excessive muscle use and/or insufficient oxygen supply to the muscle fibers, depending on factors such as metabolic demand of the task, stored energy reserves, and hydrostatic pressure and capillary bed structure. It is associated with reduced force production and slower dynamics, at least in part from reduced availability of ATP for processes related to both 88 activation and cross-bridge cycling (Barclay, Arnold et al. 1995; Allen, Lamb et al. 2008; Enoka and Duchateau 2008). The effects, however, are only qualitatively understood, hence the vague categorization of fatigue into moderate and excessive. The effects also interact with other time-varying processes such as potentiation, both of which are particularly prominent in fast-twitch muscle (Brown, Satoda et al. 1998). Because the terms of the functions in Virtual Muscle correspond to identifiable physiological processes, it should be relatively straightforward to modify their coefficients to reflect the various effects observed in association with fatigue (Enoka and Duchateau 2008), but this remains to be done. Understanding the process of onset and reversal of fatigue during physiological conditions would benefit greatly from the model presented here because the energetic cost is a measure of metabolic demand and therefore can help estimate when it would exceed chemical supplies of stored energy. The division of muscle models into slow and fast fiber types is important because they have substantially different energy rates, energy stores, metabolic processes and fatigue mechanisms. These differences have significant effects on the magnitude and nature of fatigue phenomena because they affect the rate at which stored fuel is consumed. Adaptive controllers that learn to minimize a cost-function require “knowledge of results” as a teacher, so it will be important to extend this model of energy consumption by muscle to predict metabolic products and other effects that can be sensed by the nervous system. Aerobic and anaerobic metabolism of slow- and fast- twitch muscles, respectively, have substantially different effects on CO2 production, 89 oxygen consumption, lactic acid excretion and heating. Neural sensors related to energy consumption presumably include intramuscular chemoreceptors for pH and oxygen, efference copy from the motoneurons, afferent and efferent-copy signals related to increased respiratory effort, and other manifestations of increased heat dissipation such as body temperature and perspiration. While there is substantial evidence that subjects perceive and attempt to minimize energy consumption and fatigue when performing repetitive tasks, there is also paradoxical evidence that they do not compensate for fatigue once it has occurred (Jones and Hunter 1983). The fiber-type specific model of energy consumption presented here provides a starting point for a quantitative model of the many perceived and silent consequences of muscle work. Appendix A: Virtual Muscle ‘Lumped Units’ Algorithm We have modified the Virtual Muscle continuous recruitment algorithm (Song, Raphael et al. 2008) to have sufficient modularity for the formulation of energetics and to improve accuracy in force and energy estimates without compromising its computational efficiency. The new recruitment algorithm presented here is termed ‘Lumped Units.’ The original Virtual Muscle (‘Natural Units’; (Cheng, Brown et al. 2000)) divided the muscle into discrete motor units, each of which could be recruited and frequency modulated according to Henneman’s size principle. While various exceptions to this stereotyped recruitment have been reported, it accounts well for motoneuron activity reported under dynamic locomotor conditions (Hoffer, 90 Sugano et al. 1987) similar to those obtaining in the cycle ergometry task [41] used for Validation - Effects of Kinematics. Obtaining a smoothly modulated total muscle force required a model with many summed units, which was computationally too expensive for multimuscle systems. (Song, Raphael et al. 2008) provided a lumped recruitment algorithm that allowed each muscle fiber type to be represented by a single motor unit with a smoothly modulated firing rate and a gradually increasing percentage of the PCSA representing its share of the total muscle. This reduces the number of states to be solved (from 2+3*# of motor units to 3+3*# of fiber types). Retaining force-length, activation, and recruited fractional physiological cross- sectional area (fPCSA) terms from the natural recruitment algorithm allow independent calculation of energy consumption for slow and fast fibers and to scale the tetanic enthalpy data. We here introduce a weighting function that computes an effective activation for each fiber type that more accurately reflects the sum of the nonlinear relationships relating excitation to firing rate of each motor unit. Because sarcomeres move almost homogeneously during muscle contraction and their individual contributions to total muscle force sum approximately linearly, the force- length and force-velocity relationships have identical form in the lumped model. The recruited fractional physiological cross sectional area (rF pcsa) term for each fiber type is also preserved in the lumped model, as it is important for scaling the energy consumption of each fiber type. The modifications and additions presented here do not add significantly to the computational load, so the reductions in 91 computing time reported by (Song, Raphael et al. 2008) for lumped recruitment still obtain for this new version of VM. Motor units are recruited continuously in this scheme; therefore, we modeled rF pcsa for a given fiber type as a linear function of drive that is zero when the drive is equal to threshold and equal to its F pcsa at maximal drive. <= >?@A B = C 0, D ≤ D EF B G HIJK L MNO PQ L RD − D EF B S, D > D EF B T , where i = slow, fast We chose a weighting function that modulates the activation of the lumped units such that the steady state force matches that of the discrete model. For an arbitrary drive between 0 and maximal (normalized to 1) the activation of the lumped unit overestimates the total activation of the discrete motor units because the later recruited units within that fiber-type will be firing at lower rates than the first recruited unit (see Fig. 8a). To compensate for this discrepancy across different fiber compositions (represented by parameter U V (W ) and neural drive (U), we created the weighting function shown in Fig. 8b. It is a two-part piece-wise function in which the first part is a parabola spanning from the fiber type’s stimulation threshold to the threshold of the next recruited fiber type for the slow fiber type and U r for the fast fiber type. The second part is an exponential that is defined all the way to maximal drive. X @YZ[ D, D EF \A@E = ] ^ _ X @M = 1.556D c − 1.203D + 0.8844, D EF @YZ[ ≤ D < D EF \A@E X @c = X @M R D EF \A@E S + i ∗ j1 − k N lml PQ nKJP o J p , D > D EF \A@E T 92 where i = −1.99D EF \A@E r + 1.899D EF \A@E s − 1.748D EF \A@E c + 1.186D EF \A@E + 0.1298 u @ = 0.261 X \A@E D, D EF \A@E = ] v ^ v _ X \M = w D − x y z c + 0.65, D EF \A@E ≤ D < D { X \c = X \M D { + | ∗ }1 − k N lml ~ o n , D ≥ D { T where x = 0.5859D EF \A@E + 0.3934 y = −98.82D EF \A@E € + 155.7D EF \A@E r − 85.96D EF \A@E s + 19.32D EF \A@E c − 2.618D EF \A@E + 1.023 | = 0.35 u \ = −0.268D EF \A@E + 0.2548 93 Figure 3-8: Comparison of ‘Natural’ and ‘Lumped’ recruitment schemes (a) Motor units in the ‘Natural Units’ scheme are recruited sequentially according to fiber type and size. In ‘Lumped’ recruitment, each fiber type is represented by a single motor unit whose frequency modulation is identical to the first recruited unit within the corresponding fiber type in the ‘Natural Units’ scheme. The plot at the bottom illustrates the error resulting in activation (which scales force and energy) given this approximation and highlights the need of a weighting function to capture the nonlinear effects on activation caused by sequential recruitment of discrete motor units. Weighting functions for slow and fast-twitch fibers (b). A weighting function is shown for each fiber type that matches the steady state force computed by the ‘Lumped’ model to that of ‘Natural Units.’ The weighting function is dependent on excitatory drive as well as fiber composition. Vertical lines mark the recruitment threshold of the fiber type. See text and Tables 1 and 2 for a detailed description of the functions that generate these curves. 94 The resulting steady state force prediction (<= >?@A ∗ X ∗ i� ∗ =‚ ∗ =ƒ of the new algorithm is nearly identical to that of the ‘Natural Units’ model (see Fig. 9). To match the dynamic response, we subjected the input drive to a first-order low pass filter with a variable time constant depending on whether the drive is increasing or decreasing. Without this filter the weighting function and consequently the muscle force at an intermediate activation would change instantaneously in response to a drive, which is unphysiological. To model the complex dynamics resulting from newly recruited units that are not captured by the lumped unit, we made the time constant for rising drive a function of the change in drive (U-U eff). The exact form of the function varies with fiber type composition because fiber types have rise and fall times that differ significantly. For decreasing drive, a simple time-constant irrespective of drive was sufficient. D „ …\\ = D − D …\\ † O , † O = ‡ † {B@… k NR ONO ˆnn S∗ ‰ ~LJˆ ∗M , D ≥ D …\\ † \AYY , D < D …\\ T where † {B@… = 0.38D EF \A@E c + 0.8D EF \A@E + 0.14 † \AYY = −0.32D EF \A@E r + 0.82 ∗ D EF \A@E s − 0.28D EF \A@E c + 0.014D EF \A@E + 0.09 95 Figure 3-9: Comparison of steady state response to an excitatory step The force predicted by ‘Natural’ and ‘Lumped’ recruitment schemes is shown for excitatory drives ranging from 0 to 1 (maximum). The predictions of the two schemes are nearly identical. Fig. 10 shows a comparison of the dynamic force response between ‘Natural’ and ‘Lumped’ algorithms for a wide range of stimulation frequencies and amplitudes. Fig. 10a shows the similarity in both the transient and steady state response, which can be decomposed into mean force and modulation range as shown in Fig. 10b. 96 Figure 3-10: Comparison of dynamic response to sinusoidal input Predictions of ‘Natural’ and ‘Lumped’ recruitment schemes are shown for an exemplary stimulus (a). The response is divided into a transient and steady state phase, which is divided further into mean force and modulation range components. A comparison of these components across a wide range of amplitudes and excitation frequencies is shown in b. 97 Figure 3-11: Muscle model overview Conceptual overview of functions that comprise the force and energetics models as well as their interactions. Most of the functions have a one-to-one correspondence with physiological processes underlying experimental phenomena from which the data were obtained. See text for details. A conceptual overview of VM’s force and energetics models is shown in Fig. 11. The contractile force and energy consumption of muscle can be separated into contributions from a number of physiological processes, some of which affect both formulations. The force produced by the contractile element strains the series elastic element (representing the tendon/aponeurosis) through an intermediate mass to generate a pulling force on the attached segments. The contractile force has 98 passive and active contributions. The passive portion is a result of the visco-elastic properties of stretching muscle and the elastic properties of the thick filament upon compression at short muscle lengths. The active portion can be divided into three major physiological processes: Activation-force (Af), Force-length (FL) and Force- velocity (FV). The FV relationship is mainly related to the overall strain of cross- bridges, which depends on the angle of attachment of the myosin heads onto the actin sites. The FL relationship captures the effect of myofilament overlap on force and is related to the number of cross-bridges formed. The Af relationship reflects the portion of overlapping myofilaments that bind and is dependent on firing rate, muscle length and fiber-specific properties (e.g. sag and yield). The weighting function, W, only applies to the lumped recruitment model and is used to scale the activation-force relationship of the lumped unit in order to match the total Af of a realistic ensemble of motor units. The effective frequency input (f eff) to the Af relationship is determined by calculating the stimulation frequency (f env) first, which is directly proportional to the neural drive (U) if U is greater than the firing threshold. A second-order low pass filter, whose exact form depends on fascicle length and activation, is then applied to f env to obtain f eff. Because Af and FL together provide a measure of the number of cross-bridges, they also affect the energy related to cross-bridge cycling (E xb). They are used to scale the tetanic EV relationship obtained experimentally at optimal sarcomere length. The energy related to excitation (E a) is simply a function of stimulation frequency, which is added to E xb to obtain the total energy rate during the initial phase of contraction 99 (E initial). This energy represents the amount of ATP and PCr molecules consumed and therefore determines the energy needed to resynthesize them (E recovery). The sum of E initial and E recovery is the total energy rate. Appendix B: Modeling details for dynamic knee extension task To compute metabolic energy consumption associated with the dynamic knee extension task, we built a model that mimicked basic biomechanical properties of the leg and the performance conditions of the experiment. Because the task lasted for a relatively long period of time, it was important to include the dynamics of energy consumption related to recovery in the energetics formulation (see Validation). Musculoskeletal System The musculoskeletal model of the leg consisted of two segments (thigh and shank plus foot) linked by a hinge joint at the knee. Two muscle elements were included to represent the knee extensors, the only active muscles in this task. One element corresponded to the rectus femoris muscle and the other to the vasti muscles (vastus lateralis, intermedius and medialis). The vasti muscles were lumped together because they have similar attachment points, moment arms and excitatory drive for this specific task. Rectus femoris was modeled separately because it has a longer tendon, whose compliance would affect fascicle velocity during this dynamic task. It can be assumed that they are all recruited to similar levels because the modeled task required subjects to use maximal effort. The 100 dimensions of the skeletal segments and muscle attachment points were derived from (Shelburne and Pandy 1997). All muscles attach to the same point on the shank via the patella, thus they have the same moment arm, which is virtually independent of joint angle (O'Connor 1993). Morphometric measures of the muscles such as fascicle to tendon length ratios were obtained from (Hoy, Zajac et al. 1990) and the shape of the maximal isometric torque versus joint angle was validated against human data presented in (Pincivero, Salfetnikov et al. 2004). Muscle mass was derived from data presented in (Hoy, Zajac et al. 1990; González- Alonso, Quistorff et al. 2000). See Fig. 12a for a detailed description of the musculoskeletal system. 101 Table 3-1: Table of Equations Equations Slow-twitch fibers Fast-twitch fibers ( ) + − = 1 exp ln T T r se T T se se k L L k c L F 8 . 27 T c 0047 . T k 964 . 0 T r L 8 . 27 T c 0047 . T k 964 . 0 T r L ( ) ce r ce ce ce ce pe V k L L L k c V L F η + + − = 1 exp ln , 1 1 max 1 1 1 0 . 23 1 c 046 . 0 1 k 17 . 1 1 r L 01 0. η 0 . 23 1 c 046 . 0 1 k 17 . 1 1 r L ( ) ( ) [ ] { } , 1 exp 2 2 2 2 − − = r ce ce pe L L k c L F 0 2 ≤ pe F 02 . 0 2 − c 0 . 21 2 − k 70 . 0 2 r L 02 . 0 2 − c 0 . 21 2 − k 70 . 0 2 r L ( ) − − = ρ β ω 1 ce ce L exp L FL 12 . 1 ω 30 . 2 β 62 . 1 ρ 75 . 0 ω 55 . 1 β 12 . 2 ρ ( ) ( ) ( ) [ ] ( ) [ ] ( ) > + + + − ≤ + + − = 0 , 0 , , 2 2 1 0 1 0 max max ce ce v ce ce v ce v v v ce ce ce v v ce ce ce V V b V L a L a a b V V L c c V V V L V FV 88 . 7 max − V 88 . 5 0 v c 0 1 v c 15 . 9 max − V 70 . 5 0 − v c 18 . 9 1 v c 70 . 4 0 − v a 41 . 8 1 v a 34 . 5 2 − v a 35 . 0 v b 53 . 1 0 − v a 0 1 v a 0 2 v a 69 . 0 v b ( ) − − = f n f f eff ce ce eff slow n a Yf V L f Af exp 1 , , ( ) − − = f n f f eff ce eff fast n a Sf L f Af exp 1 , − + = 1 1 1 0 ce f f f L n n n 56 . 0 f a 1 . 2 0 f n 5 1 f n 56 . 0 f a 1 . 2 0 f n 3 . 3 1 f n ( ) ( ) Y Y ce Y T t Y V V exp c t Y − − − − = 1 1 & 35 . 0 Y c 1 . 0 Y V 200 ) (ms T Y – – – 102 ( ) ( ) S S eff T t S a f t S − = , & , ( ) ( ) ≥ < = 1 . 0 , 1 . 0 , 2 1 t f a t f a a eff S eff S S – – – 76 . 1 1 S a 96 . 0 2 S a 43 ) (ms T S ( ) ( ) ( ) f env ce env T t f t f L f t f int int , , − = & ( ) ( ) ( ) f eff ce eff T t f t f L f t f − = int int , , & ( ) ( ) < + ≥ + = 0 , 0 , 4 3 2 2 1 eff ce f f eff env f ce f f f L Af T T f t f T L T T & & 3 . 34 ) ( 1 ms T f 7 . 22 ) ( 2 ms T f 0 . 47 ) ( 3 ms T f 2 . 25 ) ( 4 ms T f 6 . 20 ) ( 1 ms T f 6 . 13 ) ( 2 ms T f 2 . 28 ) ( 3 ms T f 1 . 15 ) ( 4 ms T f U eff eff T U U U − = & , < ≥ = − − eff fall eff T U U rise U U U T U U e T T rise eff ) 1000 ln( ) ( 14 . 0 8 . 0 38 . 0 2 + + = fast th fast th rise U U T 09 . 0 014 . 0 28 . 0 82 . 0 32 . 0 2 3 4 + + − + − = fast th fast th fast th fast th fall U U U U T 01 . 0 , ), ( 1 , 0 = > − − ≤ = slow th slow th slow th slow th slow pcsa slow th slow pcsa U where U U U U U F U U rF ) 8 . 0 ( * , ), ( 1 , 0 = = > − − ≤ = r slow pcsa fast th fast th fast th fast th fast pcsa fast th fast pcsa U F U where U U U U U F U U rF > − + = < ≤ + − = = − fast th U U fast th s s fast th slow th s fast th slow U U e A U W W U U U U U W U U W s fast th , 1 ) ( , 884 . 0 20 . 1 56 . 1 ) , ( 1 2 2 1 τ ≥ − + = < ≤ − = = − r U U fast th f f r fast th f fast th fast U U e D U W W U U U C B U W U U W f r , 1 ) ( , ) , ( 1 2 1 τ 13 . 0 19 . 1 75 . 1 90 . 1 99 . 1 2 3 4 + + − + − = fast th fast th fast th fast th U U U U A 26 . 0 = s τ 39 . 0 59 . 0 + = fast th U B 1.02 + 2.62 - 19.32 + 85.96 - 155.7 + -98.82 2 3 4 5 fast th fast th fast th fast th fast th U U U U U C = 35 . 0 = D 25 . 0 27 . 0 + − = fast th f U τ V e e V e V e V L L f f E ce ce tet initial − + + = = = 4 3 2 1 0 * 2 ^ * ) , , ( 6 . 76 1 − e 792 2 − e 124 3 e 72 . 0 4 e 145 1 e 3322 2 − e 1530 3 e 45 . 1 4 e Energetics constants 33 . 0 a 5 . 1 R 25 . 0 M 33 . 0 a 1 R 25 . 0 M 103 Table 3-2: Symbols and Definitions Symbols Definitions D Activation input (0-1) = >?@A B Fractional PCSA of i th fiber type (0-1) < = >?@A B Recruited fractional PCSA of i th fiber type (0-= >?@A B ) D E F B Recruitment threshold for i th fiber type (0-1) D { Fractional activation level at which all fiber types for a given muscle are recruited (0-1) D …\\ Effective drive, an intermediate muscle activation signal simulating calcium dynamics (0-1) X B Weighting function that determines a lumped unit’s (fiber type) effective i� (0-1) = Muscle maximal tetanic force (N) ‚ ?… Optimal fascicle length (cm) ‚ @… Optimal tendon length (cm) = Š @… Series elastic element (tendon) force (F 0 ) = Š >… M Stretching contractile passive element (fascicle) force (F 0 ) = Š >… c Compressive contractile passive element (fascicle) force (F 0 ) = Š ?… Active contractile element force (F 0 ) ‚ Š @… Tendon length (L se0 ) ‚ Š ?… Fascicle length (L ce0 ) ‚ Š ?… ‹AŒ Maximum fascicle length of the muscle at its maximum anatomic musculotendon length (L ce0 ) ƒ Š ?… Fascicle velocity (L ce0 /s) =‚ B Force-length function of i th muscle fiber type =ƒ B Force-velocity function of i th muscle fiber type i� B Activation-frequency relationship of i th fiber type � Yielding factor for slow fiber type Ž Sagging factor for fast fiber type � …�� B Firing frequency input to 2 nd order excitation dynamics of i th fiber type (f 0.5 ) � E…E Tetanic motor unit firing rate � B�E B Intermediate firing frequency of 2 nd order excitation dynamics of i th fiber type (f 0.5 ) � …\\ B Effective firing frequency of i th fiber type (f 0.5 ) ‘ A Energy rate related to excitation (mW) ‘ Œ’ Energy rate related to crossbrige cycling (mW) ‘ B�BEBAY Energy rate related to ATP and PCr consumption, E a + E xb (mW) ‘ {…?Z�…{“ Energy rate related to ATP and PCr recovery (mW) ‘ EZEAY Total energy rate per unit mass (mW/g wet ) ” E a partition relative to E tot for the isometric condition • Ratio of recovery to initial energy (E recovery / E initial ) – Muscle mass unit conversion factor (g dry /g wet ) i—˜ …\\ < = >?@A B *W*Af (0-1) Note: Top barx denotes the normalized variable x (forces by maximum isometric tetanic muscle force F 0, lengths and velocities by optimal fascicle length or optimal tendon length [L ce0 or L se0 ]). 104 Stimulation Parameters Stimulation parameters were chosen to model muscle excitation and kinematic conditions of the experiment in (González-Alonso, Quistorff et al. 2000). The movement performed by the subjects was smooth and periodic, so it was modeled as a sinusoidal trajectory. This kind of maximal performance generally requires “bang-bang” controls in which all and only those muscles that can contribute positive work at a given phase of the task are maximally recruited (Zajac 1985). Therefore we activated both modeled muscles at 80%MVC (see Validation regarding MVC) during the first quarter of the movement cycle, which corresponded to the acceleration phase of the knee extension movement. Refer to Fig. 12b for a graphical depiction of the stimulation parameters. Dynamics of Recovery Energy As mentioned in Model Design and Rationale, every ATP and PCr molecule expended must be replenished via metabolic processes. These metabolic processes consume energy (E recovery) and last on the order of a few minutes. The hydrolysis of ATP and PCr molecules is closely related to the E initial term in the model and can therefore be used to drive the formulation of E recovery. The formulation used for this task is based on the assumption that the recovery system behaves linearly for the range of ATP and PCr deficits occurring in the subjects’ muscles. This allows complete characterization of the system’s behavior based on its impulse response, 105 which is derived for slow and fast-twitch fiber types from (Leijendekker and Elzinga 1990) during brief tetanic contractions that activate only aerobic metabolism in both fiber types (see Validation regarding Effects of Kinematics). Because the duration of E initial measured in this experiment was small compared to the duration of E recovery, it was assumed that the E recovery reported was the response to an impulse weighted by the total energy consumed in phase with the contraction. Therefore, the unit impulse response incorporated into the model (shown in Fig. 13) is the measured trajectory of E recovery divided by the total energy consumed during the contraction. 106 Figure 3-12: Model of the dynamic knee extension task Musculoskeletal model of the knee (a) Two separate Virtual Muscle elements are included representing the rectus femoris and vasti muscles, whose properties are listed in the table. Stimulation parameters (b). The knee was constrained to follow a sinusoidal trajectory ranging from 80 to 170 degrees of extension. Both muscles were excited at 80% of their maximal drive during the first quarter of each movement cycle (acceleration phase). Stimulation parameters are shown for the first three seconds of the simulation to illustrate their relative phasing. The actual duration of the exercise was 180 seconds. 107 Figure 3-13: Characterization of the ATP/PCr recovery system The system accepts the amount of energy consumed due to ATP/PCr turnover (ΔE initial) as an input and generates the energy rate corresponding to the metabolic processes needed to replenish the expended ATP/PCr. Traces at bottom were digitized and rescaled from Leijendekker and Elzinga (1990), who measured E recovery following brief tetanic stimuli. These curves were convolved with the incremental increases of the integral of E initial to capture the dynamics of E recovery. See text for details. Acknowledgment We thank Norman Li for technical assistance with implementing the model in software. 108 Chapter 4: Spinal Circuitry Physiology and Modeling George A. Tsianos Neurophysiology The role of reflexes in the coordination of muscles and movements dates back to Sir Charles Sherrington, whose seminal work was published in the Integrative Action of the Nervous System in 1906. The book is derived from a set of ten lectures (Silliman lectures) he delivered at Yale University two years earlier. It mainly addresses the functional significance of the neuron doctrine as supported by Ramon y Cajal’s extensive neuroanatomical studies (Ramon y Cajal 1907). The notion that the nervous system consists of discrete elements, i.e. neurons with specific polarity prompted Sherrington to investigate the principles that govern how the information encoded by their electrical impulses is integrated. Sherrington viewed sensorimotor integration from the standpoint of a reflex, which he defines as the fundamental unit of reaction in the nervous system relating activity between receptor (e.g. muscle spindle) and effector (e.g. skeletal muscle). He asserted that a given reflex consists of an intermediate element between receptor and effector that transmits the activity. Because the conduction delay of reflexes was typically larger than the conduction delay of neuronal processes alone, 109 he postulated that reflex pathways consist of breakpoints (which he termed synapses) that interrupt the conduction, hence contributing to the additional delay observed in his experiments. Therefore, the existence of multiple synapses within a given reflex arc implied the existence of intermediate neurons, presently called interneurons, between receptors and alpha motoneurons that make up the “final common pathway” to the muscles. Interneurons within the spinal cord comprise a significant component of the proposed work, as the main focus is to study the sensorimotor processing occurring at this level and its implications for control by the brain. As recognized by Sherrington, the notion of simple reflexes (often consisting of several interneurons) provides a convenient framework for investigating sensorimotor integration, but should be extended to include interactions among them to reveal the true nature of the processing. Reflexes neither operate nor can be understood in isolation because their activity is highly coupled via interneuronal connections and precisely coordinated when performing purposeful behavior. Consequently, understanding their role, as well as sensorimotor integration as a whole, necessitates knowledge of the connectivity of interneurons. Interneurons identified via electrophysiology Over the last century, the spinal circuitry has been investigated in great detail in neuroanatomical and electrophysiological studies. Electrophysiological techniques have been the most practical means for studying the functional connectivity of 110 interneurons (see Pierrot-Deseilligny and Burke 2005 for review). These techniques are performed by systematically controlling afferent signals while observing their effects on EMG, namely enhancement/depression of background activity as well as its timing. As a result of combined efforts over the last century, primarily electrophysiological, a significant portion of the highly intricate and profuse neural circuitry of the spinal cord has been identified. Below is a list of the classical interneuronal pathways that have been identified using electrophysiological techniques (summarized in Figure 1) 1 : 1 The remainder of this subsection up until “Other methods to identify interneuronal circuits” is an excerpt from Chapter 7, which is a published article that has been reprinted with permission. See beginning of Chapter 7 for details. 111 Figure 4-1: Classical interneuronal pathways Classical interneuronal pathways drawn from the perspective of a single muscle. Projections from self (HOM) as well as synergist (SYN) and antagonist (ANT) muscles are shown. Propriospinal pathway: The proprioceptive sensory outputs (muscle spindle primary Ia and Golgi tendon organs Ib) excite the priopriospinal interneurons (Malmgren et al. 1988; Gracies et al. 1991; Burke et al. 1992). These directly excite the homonymous motoneurons, and also form the following connections with heteronymous motoneurons: excitatory for the synergists and inhibitory for the antagonist muscles. Monosynaptic Ia excitation pathway: Muscle spindle primary afferents excite the motoneurons monosynaptically (Malmgren et al. 1988a). These afferents also 112 excite the heteronymous motoneurons of synergist muscles (Lloyd 1946; Meunier et al. 1993). Reciprocal Ia inhibition pathway: The Ia inhibitory interneuron form three characteristic connections: 1) monosynaptic input from Ia afferents, 2) inhibitory projections to antagonist and synergist motoneurons (Tanaka 1974; Pierrot- Deseilligny et al. 1981; Crone et al. 1987), and 3) disynaptic recurrent inhibition (Renshaw pathway) from homonymous motoneurons (Hultborn et al. 1971). The Ia interneurons were themselves inhibited by the corresponding antagonist Ia interneurons (Hultborn et al. 1976). Renshaw inhibition pathway: Renshaw interneurons form recurrent connections with homonymous motoneurons. They are also excited by motoneurons of synergist muscles (Eccles et al. 1954, 1961). They inhibit antagonist Renshaw interneurons (Hulborn et al. 1979; Ryall et al. 1981). Ib inhibitory pathway: The Ib interneuron receives excitatory input from Golgi tendon organs. The dominant effects of the Ib-inhibitory interneuron are inhibition of homonymous motoneurons, excitation of antagonist motoneurons and inhibition of synergist motoneurons (Eccles et al. 1957). Interneurons and connections listed below were also identified but were excluded from the study to limit the complexity of the model and to minimize the number of arbitrary decisions required to model poorly characterized pathways. Because the model of the spinal cord presented here has proven to be surprisingly tractable computationally, it would be feasible to add them in the future, but this 113 would be useful only to account for behaviors that the model cannot perform well. As described below, the model accounted well for all motor tasks attempted so far. • All cutaneous input was omitted because the tasks to be modeled can be performed without it and because the details of these oligosynaptic circuits are poorly described. • The excitatory and inhibitory group II interneurons as well as all group II projections from the muscle spindles were omitted because their heteronymous connectivity is not well-characterized (Jankowska 1992; Lundberg 1987). Furthermore, the excitatory and inhibitory group II pathways parallel the Ia pathways to a great extent, suggesting a functional similarity between them (Lundberg 1987). The variable gamma-static and gamma-dynamic gains in our model allow for the adjustment of the relative sensitivity of Ia fiber activity to muscle length and velocity, so it captures to some extent the group II contribution to movement. • The known, but poorly defined pattern of heteronymous Ia projections to Ib interneurons (Jankowska 1992) was omitted. An important property of this convergence is that it provides a reflexive adjustment of the firing sensitivity of the Ib interneuron with respect to the phase of the movement. This may be especially important for simulating more complicated tasks and we plan to incorporate some of these projections in future models. 114 • The disynaptic inhibition of propriospinal interneurons was omitted. The apparently diffuse connectivity of these cutaneous and proprioceptive inhibitory effects (Alstermark 1984) suggests that they may be responsible for setting the bias of the propriospinal interneurons, while the more specific excitatory projections (Ilert 1978; Malmgren 1988b) seem more likely to mediate functional relationships among the muscles and their modulation during the dynamic phase of the movement. Furthermore, the feedback inhibitory interneurons are more sensitive to input from cutaneous fibers (Malmgren 1988b; Alstermark 2007), which could be especially important for tasks that include state transitions, such as between reach and grasp, which we have not modeled. Commisural interneurons were excluded from our model principally because the details of their connectivity are largely unknown and they appear not to be especially important for the tasks that we are simulating. Their bilateral projections in the spinal cord, mostly to other segmental interneurons, suggest that they may be primarily responsible for coordinating motion of contralateral limbs as in locomotion (Jankowska 2008). Other methods to identify interneuronal circuits It is important to note that a major drawback of electrophysiological techniques is their poor ability to characterize polysynaptic pathways. Even if one of the neurons in the polysynaptic chain is significantly hyperpolarized via descending 115 control, the whole chain will be invisible to the experimenter. This makes the reflexes highly dependent on the conditions of the experiment as well as on the physiological state of the subject. Furthermore, afferent collaterals may activate several reflex arcs in parallel with similar latencies, making them indistinguishable in the recorded EMG signal. Consequently, electrophysiological studies focus mostly on “last-order neurons” that make up the shortest paths from afferents to motoneurons. Fluorescent dye and viral labeling have been used to study spinal connectivity, but with considerably less success. Intracellular injections of fluorescent dye have been used to study the morphology and location of interneurons and afferent projections, which has led to discovery of interneurons with input/output properties that confirmed results from electrophysiological studies (e.g. Renshaw: Jankowska and Lindstrom 1971; Ia inhibitory: Jankowska and Lindström 1972). The multimodal and profuse input to Ib interneurons is also evident through this method (Brown 1981). Although this technique helps identify the location and afferent inputs of various interneurons, it is limited by the dye’s ability to diffuse reliably across synapses over long distances. Transneuronal viral labeling methods are more attractive for visualizing large neural networks because the labeling agent replicates inside the nucleus of each host cell, effectively amplifying its effect as it diffuses across a neural network. This advantageous feature of the method can help identify connectivity among motor nuclei that reside in different spinal segments. However, this technique generally has poor selectivity, staining large networks 116 whose precise connections on the synaptic level are impractical to visualize and whose constituent interneurons belong to many functional classes. Genetic labeling is a promising technique that holds great potential for labeling interneurons within a particular class, regardless of their location in the spinal cord. This method is based on the finding that interneurons are distinguishable based on the transcription factors and genes they express, which can be labeled early in development and visualized as the spinal cord matures (Matise and Joyner 1997; Jessell 2000; Lee and Pfaff 2001). To date, however, interneurons are merely divided into four broad classes (V0-V3) that do not correspond to the classical interneuronal types whose functional properties are known. Confocal calcium imaging holds potential in chararacterizing interneuronal morphology while also providing insight into the function of various interneuronal types (Ritter, Bhatt et al. 2001). Specific classes of interneurons can be labeled depending on the injection site of the calcium indicator and their morphology can be visualized with very high spatial resolution. The intensity of the fluorescence correlates with the calcium-mediated neuronal activity, indicating the activation level of labeled circuits. Although this technique has the unique potential of simultaneously informing about structure and function, it is severely limited by the susceptibility of the fluorescence signal to optical occlusions. This is why it has been applied most successfully to zebrafish larvae, a freshwater fish with a transparent body. 117 BOLD fMRI is a functional imaging tool that is less susceptible to interference from intermediate tissue but has relatively poor spatial and temporal resolution (Stracke, Pettersson et al. 2005). This technique detects the portion of the spinal cord that is active during a behavior e.g. the specific segment and gross region within the cross-section of that segment; however, its resolution is not adequate for discerning which classes of interneurons are active during any phase of the behavior. 118 Chapter 5: Previously Developed Models of Components George A. Tsianos The interneuronal pathways summarized are analogous to the reflex arcs defined by Sherrington. To generate purposeful behavior, the brain must learn to coordinate their activity, which ultimately produces the appropriate patterns of muscle activation. The complex physiology of muscle and spinal circuitry makes it formidable to intuit how they may be controlled to perform even the simplest motor tasks. The highly coupled non-linear dynamics of muscle contraction, for example, make it difficult to infer muscle force output based on knowledge of the excitation signal. The intricate and diffuse connectivity of the spinal cord further complicates the control problem as it transforms the descending commands in a non-intuitive fashion depending on proprioceptive feedback, which is modulated by muscle kinematics and force. Quantitative models of well-known components of the sensorimotor system capture their complex properties and enable sensible predictions about their interactions and behavior under untested conditions (Valero-Cuevas, Hoffmann et al. 2009). Because knowledge of the structure and function of the brain is relatively incomplete, it appears most sensible to study its role in motor control from the perspective of what kinds of computations would be sufficient for controlling a 119 realistic model of the lower motor system (spinal cord plus musculoskeletal system). The soundness of such bottom-up analysis is fundamentally limited by the accuracy to which the constituent elements are modeled. This proposal therefore includes, where possible, models of the relevant physiology that capture most of the known qualitative behavior. Muscle force and energetics Classical Hill-type muscle models (see Zajac 1989 for review) were not used in this study since they are typically inaccurate in predicting muscle force during physiological use (Perreault, Heckman et al. 2003). They do not capture fiber-type specific phenomena such as sag and yield that significantly affect activation of the muscle. Moreover, most of them do not account for the qualitative effects of sarcomere length on muscle activation, or the length dependence of the force- velocity relationship. A popular model proposed by Fuglevand et al. (1993) is essentially one of motor unit recruitment that predicts isometric force and associated EMG based on common drive to the motor pool. This model is not suitable for the proposed work as it does not capture the effects of sarcomere kinematics on muscle force and it also does not include tendon/aponeurosis, which affects the motion of the sarcomere, hence muscle force and proprioceptor discharge. This model also relies on assumptions of maximum discharge rates and range of recruitment modulation that 120 appear to have a significant effect on its predictions (Keenan and Valero-Cuevas 2007). Zahalak’s model (1981) is an approximation and computationally efficient version of Huxley’s framework for muscle contraction, which explicitly accounts for the biochemical state of the entire population of crossbridges. Although the model captures some behavior e.g. yielding reasonably well, it relies on tuning of parameters whose values cannot be readily identified experimentally. It requires, for example, knowledge of rate constants that describe the dynamics of individual crossbridges and the amount of calcium released per excitatory pulse, which affects the energetics formulation in particular (Ma and Zahalak 1991). With a significant amount of unknown parameters, users run the risk of over-fitting them to a relatively small class of scenarios, thus sacrificing generalization across untested ones. It is also worth noting that the framework of the model is based on assumptions that may be acceptable in some scenarios but not in others, making the parameter values used to fit the experimental data compensate for the discrepancy to some extent, and therefore, losing association with their presumed role. The ideal mathematical model of muscle force and energetics would have terms that correspond to all of the underlying physiological processes. This is not desirable because many of them may have negligible effects on force and energetics and some of the necessary experiments would simply be impractical. Virtual Muscle is a muscle model that consists of a set of modules corresponding to physiological processes that have significant effects on force and energetics, and whose behavior 121 was characterized rigorously using practical experimental techniques. The model computes muscle force and energy consumption based on an excitation input that mimics the common drive to a given motor pool and kinematic state defined by the motion of the attached segments and interactions with tendon/aponeurosis. The model’s response also depends on morphometric parameters such as mass, optimal fascicle length and fiber composition. The force produced by the contractile element (CE) strains the series elastic element (representing the tendon/aponeurosis) through an intermediate mass to generate a pulling force on the attached segments (Figure 2). The contractile force has passive and active contributions. The passive portion is a result of the visco-elastic properties of stretching muscle (PE 1 in Figure 2) and the elastic properties of the thick filament upon compression at short muscle lengths (PE 2). The active portion can be divided into three major physiological processes: Activation-force (Af), Force-length (FL) and Force-velocity (FV). A detailed description for each of these processes can be found in journal article I. 122 Figure 5-1: The schematic on the left depicts the various components of the muscle model and their interactions (Song et al. 2008) See text for details. The SIMULINK implementation of the model is shown to the right, which converts excitation and musculotendon kinematics into muscle force (Fm), fascicle length (Lce), and metabolic energy consumption (E_slow and E_fast for slow and fast-twitch fibers, respectively). The Lce output is used to drive the spindle model (see Chapter 2). Muscle spindle The muscle spindle model (Mileusnic, Brown et al. 2006) estimates group Ia and group II afferent responses based on fascicle kinematics and fusimotor drive. There are two types of fusimotor drive (gamma static and gamma dynamic) that affect the firing sensitivity of the spindle’s constituent fibers (bag1, bag2, and chain). Bag1 fibers are sensitive to both position and velocity, and respond to both gamma static and gamma dynamic input, while bag2 and chain are primarily length-dependent, and respond only to gamma static input. All three fibers contribute to the Ia response in a nonlinear fashion. The bag2 plus chain response is summed nonlinearly with the bag1 response such that the total afferent response is always greater than both bag1 and bag2 plus chain responses, but less than their sum (see 123 Figure 3). This type of effect has been well documented, and is referred to as partial occlusion. In fact, one of the unique features of this model (unlike for example Lin and Crago 2002; Maltenfort and Burke 2003) is that it captures this phenomenon as well as the underlying dynamics of fusimotor activation. It is, therefore, especially accurate in predicting muscle spindle response under combined fusimotor control. Another distinguishing feature of the model is that it incorporates more realistic representation of intrafusal mechanics. The model has been validated over a wide range of physiological conditions and it is likely that it will generalize well even across untested conditions given its structure-based formulation. Figure 5-2: Spindle model Anatomical sketch of the spindle and block diagram of the corresponding model. 124 Golgi tendon organ The role of Golgi tendon organs (GTOs) in motor control has been studied over several decades, and in particular, their ability to encode muscle force has been debated. The debates stem mainly from the fact that experimentalists monitor activity from a small percentage of the total amount of GTOs within a given muscle under conditions that are unphysiological (Houk and Henneman 1967; Gregory and Proske 1979; Crago, Houk et al. 1982). Mileusnic and Loeb (2006) shed new light on this debate by developing a structural-based model of a single GTO that accounts well for all qualitative features of its behavior such as non-linear summation of effects between the various motor units that innervate the GTO (other features include self- and cross-adaptation). Such a model, used in conjunction with models of motor unit recruitment and force, enables a strong prediction of the actual relationship between muscle force output and the complete ensemble of GTO responses. Mileusnic and Loeb (2009) performed this type of analysis using Virtual Muscle (Cheng, Brown et al. 2000): a validated model of muscle under physiological recruitment. The muscle force vs. ensemble discharge relationship that emerged from this modeling approach was virtually static, and could be described by a two- part piece-wise function to account for the relatively higher sensitivity of the GTO response at lower levels of force (see Figure 4). 125 Figure 5-3: Golgi tendon organ model Modeled relationship between ensemble GTO activity and whole muscle tension. Spinal cord Models of the spinal circuitry that are reported in the literature typically involve only a subset of interneuronal classes and are wired among muscles that possess simple antagonist relationships. McCrea and Rybak (2008) published a relatively detailed model of the spinal cord that coordinates muscle activity in locomotion between the broadly defined extensor and flexor groups. It includes a small subset of the identified pathways (monosynaptic Ia, Ia inhibitory, and Renshaw) and a complex set of other pathways that are presumed to exist, based on high-level experimental phenomena e.g. independent control of timing and magnitude of muscle activity, independent modulation of timing and activity via proprioceptive feedback, etc. Van Heijst et al. (1993) also modeled the spinal circuitry (extension of the FLETE model; Bullock and Grossberg 1989) between two purely antagonist 126 muscles. They too include only the monosynaptic Ia, Ia inhibitory, and Renshaw pathways. Bashor (1998) similarly modeled the spinal circuitry between an antagonist muscle pair like in the studies above, but also included the Ib pathway. This was a large-scale effort that modeled individual neurons rather than populations. Unlike previously published models, the one incorporated in the proposed study includes a significantly larger portion of the known interneuronal pathways, which are described in detail in the section titled “Interneurons identified via electrophysiology.” This model is termed spinal-like regulator (SLR). It includes antagonistic as well as synergistic connectivity, enabling simulation of more realistic musculoskeletal systems with muscle pairs whose functional relationships vary with respect to the motor task. 127 Chapter 6: The Relationship between the Spinal Cord and other Levels of the Sensorimotor System George A. Tsianos The spinal cord acts as an integrating center that mixes descending commands with proprioceptive feedback to generate motor output. Its intrinsic properties, along with those of the musculoskeletal system, largely determine the influence of descending commands on the motor output. From this perspective, a question that naturally arises is: what commands are needed from the brain to get the spinal cord plus musculoskeletal plant to generate the appropriate behavior? The sensorimotor system is “redundant” on several levels, which makes it difficult to predict the neural strategy that is most physiological. For example, many patterns of muscle activation can produce a desired kinematic performance e.g. with high or low levels of cocontraction. Further upstream, multiple combinations of spinal circuits can be elected to generate a given set of muscle activations. It is also plausible, given the diffuse and overlapping nature of descending projections, that many sets of cortical commands can activate spinal circuitry in the same manner. A model constructed by Loeb and colleagues (Loeb, He et al. 1989; Loeb, Levine et al. 1990; He, Levine et al. 1991) shed new light on the redundancy problem and on how optimal control principles may be applied to reveal emergent properties of 128 complex systems. They simulated a task involving posture regulation of the cat hindlimb using a fairly realistic representation of the musculoskeletal system and feedback gain matrices that resembled the types of feedback mediated by interneurons in the spinal cord. Linear quadratic regulator design was used to compute the optimal gains, based on a cost function that penalized for deviations from the unperturbed state and effort. Surprisingly, the model was able to predict the polarity of afferent feedback projections such as homonymous excitation from muscle spindle Ia endings and inhibition from Golgi tendon organs and Renshaw cells. Additionally, individual heteronymous projections, just like in the biological system, were not as strong, but because any given muscle received many of them, they were able to influence the motor output considerably. In fact, removing even a subset of these projections resulted in significant degradation of performance, highlighting their important physiological role. This study also provided insight into task parameters that are considered by the nervous system. It was shown, for example, that the full state feedback control strategy that penalized moderately for effort produced the most physiological reflex patterns. This technique was also instrumental in showing that apparent redundancies in overactuated limbs (10 muscles controlling 3 joints in this case) are not present when considering the actual requirements of the task (regulating muscle stiffness and effort versus regulating posture only), as evidenced by the existence of unique and substantially different solutions for each set of performance criteria. 129 Hierarchical architecture was later applied to sensorimotor systems to obtain insight into what aspects of the behavior reflect control strategy versus emergent properties of lower motor systems e.g. inherent stability of muscle. The framework of such modeling was first described in Loeb et al. (1999) and consisted of computational elements that corresponded to the brain, spinal cord and plant (musculoskeletal system plus environmental interactions). The brain was modeled as a task planner that defines the motor task, and an adaptive controller that learns the appropriate commands for achieving it. The spinal cord functioned as a programmable regulator that integrated multimodal afferent feedback and descending commands to generate alpha motoneuronal activity, which excites the muscles. The musculoskeletal plant was excited by these signals as well as by interactions with the environment. The proprioceptive information generated by the muscles not only modulated neural activity in the spinal cord, but was also received by the brain module to estimate actual performance and drive the learning process of the adaptive controller. It was shown that the contribution of each level of the hierarchy depended on how its intrinsic properties affected task performance. High reflex gains in the spinal cord, for example, were favorable for resisting sinusoidal perturbations of relatively low frequency. For higher frequency perturbations, however, a coactivation strategy was more favorable because resistance to the perturbation was predominantly contributed by the intrinsic properties of muscle, which provide a faster response (virtually instantaneous) than reflexes whose total delay, 130 including neuronal conduction and dynamics of muscle activation, may cause destabilizing effects. Even this strategy, however, was not favorable if energetics were considered. Therefore, the control strategy adopted by the nervous system was shown to be highly sensitive to the intrinsic properties of the lower systems and the performance criteria of the task, whether they are specified explicitly, e.g. follow a kinematic trajectory, or implicitly, e.g. ensure that the trajectory is stable. In fact, keeping the performance criteria constant and gradually adding more realism, i.e. noise and output saturation, to the muscle spindle significantly altered the solution space, and thus, the control problem the brain had to solve. Framing motor control as a hierarchy enables the use of a bottom-up approach to studying the types of computations performed by higher levels. This is attractive for the motor system because the structure and physiology of the lower systems (spinal cord plus musculoskeletal plant) is well defined. More modular and complete models of spinal circuitry and muscle have been developed that can be combined to study more complex systems (see Chapters 7-10). Collectively, they make up a more accurate representation of the lower motor system, therefore, enhancing the validity of hypotheses regarding the higher-level system, i.e. brain, that controls it. In this research, the general approach for studying the intrinsic properties of the lower motor system was to model the brain in the simplest possible manner and investigate how much of the details of particular motor tasks are attributable to the properties of the spinal circuitry plus musculoskeletal system. 131 Chapter 7: Spinal-Like Regulator Facilitates Control of a Two Degree-of-freedom Wrist Giby Raphael, George A. Tsianos, Gerald E. Loeb © 2010 Journal of Neuroscience, Society for Neuroscience. Reprinted with permission from: Raphael, G., Tsianos, G. A., & Loeb, G. E. (2010). Spinal-Like Regulator Facilitates Control of a Two-Degree-of-Freedom Wrist. J Neurosci, 30(28), 9431-9444. Preface The following article, published in the Journal of Neuroscience, focused on the intrinsic properties of the lower motor system in the context of wrist motion. Descending commands from the brain are modeled as unmodulated step functions that act on muscles indirectly through interneurons. A very simple learning algorithm was used to determine the appropriate amplitudes of each of 184 commands. Given the simple model of the brain, the system was able to rapidly learn complex motor tasks whose specific features have been often assumed to be directly controlled by the brain. Model tasks included stabilizing response to external force perturbation, rapid movement to a position target, isometric force to a target force level, and adaptation to viscous curl fields. 132 Contributions of the Authors Giby Raphael, George Tsianos and Gerald Loeb wrote the manuscript. Giby Raphael and Gerald Loeb designed the SLR for the wrist and developed modeling experiments. George Tsianos assisted with the physiological justification of the SLR, and analysis of results. George Tsianos and Gerald Loeb designed a servo-controller for the wrist musculoskeletal system to compare its performance with the SLR. Abstract The performance of motor tasks requires the coordinated control and continuous adjustment of myriad individual muscles. The basic commands for the successful performance of a sensorimotor task originate in “higher” centers such as the motor cortex, but the actual muscle activation and resulting torques and motion are considerably shaped by the integrative function of the spinal interneurons. The relative contributions of brain and spinal cord are less clear for reaching movements than for automatic tasks such as locomotion. We have modeled a two- axis, four-muscle wrist joint with realistic musculoskeletal mechanics and proprioceptors and a network of regulatory circuitry based on the classical types of spinal interneurons (propriospinal, monosynaptic Ia- excitatory, reciprocal Ia- inhibitory, Renshaw inhibitory and Ib-inhibitory pathways) and their supraspinal control (via biasing activity, presynaptic inhibition and fusimotor gain). The modeled system has a very large number of control inputs, not unlike the real spinal cord that the brain must learn to control to produce desired behaviors. It was 133 surprisingly easy to program this model to emulate actual performance in four very different but well-described behaviors: 1) stabilizing responses to force perturbations; 2) rapid movement to position target; 3) isometric force to a target level; 4) adaptation to viscous curl force fields. Our general hypothesis is that, despite its complexity, such regulatory circuitry substantially simplifies the tasks of learning and producing complex movements. Introduction Extensive research of the past century led to the modern view of the spinal cord as a center for sensorimotor coordination rather than just a relay from brain to muscles. For example, locomotion can be produced and controlled entirely by the spinal cord and the peripheral sensorimotor apparatus by means of a central pattern generator (CPG) and associated circuitry responsible for reflexive adjustments to various conditions such as loads, obstacles and perturbations (McCrea and Rybak 2008). Various descending pathways from the cerebral cortex, cerebellum, vestibular apparatus, brainstem, etc., modulate the function of the intrinsic spinal circuitry rather than controlling directly the highly phasic activity of the individual muscles responsible for the movement (Lavoie, McFadyen et al. 1995; Ivashko, Prilutsky et al. 2003). By contrast, studies of arm and hand movements in primates have focused largely on correlating the activity of populations of neurons in motor cortex with measureable features of the concomitant motor behavior. Inferring causation, 134 however, requires an understanding of the intervening neural circuitry and musculoskeletal mechanics (Loeb, Brown et al. 1996). Various studies have shown that the indirect pathway via subcortical or spinal interneuronal systems contribute the majority of inputs to forelimb motoneurons (Gelfan 1964; Alstermark and Sasaki 1985; Kitazawa, Ohki et al. 1993). These spinal interneurons receive inputs from the pyramidal tract (Isa, Ohki et al. 2007) and also convergent input from many modalities and sources of primary somatosensory afferents (as well as extrapyramidal descending activity) and they exert divergent excitatory and inhibitory influences on many motoneuron pools (McCrea 1986; Fetz, Cheney et al. 1989). Direct corticomotoneuronal projections are largely absent in opossums, rodents, cats and lower primates (Rathelot and Strick 2009) that are capable of fairly sophisticated, voluntary limb movements for prey-catching and/or food- handling. Over the past 20 years, there have been several attempts to model the lower sensorimotor system for voluntary movement, but they were usually restricted to a hinge joint operated by an antagonist muscle pair and they usually incorporated only a small subset of the known spinal circuitry (Feldman 1966; Bullock and Grossberg 1992; Bullock, Grossberg et al. 1993; Bashor 1998; van Heijst, Vos et al. 1998; Ostry and Feldman 2003; Lan, Li et al. 2005; Maier, Shupe et al. 2005). Loeb et al. (1990) introduced linear quadratic regulator design to find optimal solutions for reflex gains in a multisegment limb model (He, Levine et al. 1991), but there was 135 no way to reconcile these gains with the actual interneuronal elements of the spinal cord. The model of spinal circuitry presented here appears to be the first to consider the shifting patterns of muscle coordination needed to operate multi-muscle, multi- degree-of-freedom linkages in a variety of complex sensorimotor behaviors described in the experimental literature. The model is called a spinal-like regulator (SLR) because it embodies the multi-input, multi-output connectivity and gain features of an engineering regulator (Loeb et al., 1990; He et al., 1991). The most salient emergent property of the whole system is that despite the complexity of the available circuitry, many different combinations of command signals result in generally stable and desirable output behaviors. Methods Biomechanical model The biomechanical model (Figure 1) was implemented in MATLAB. The human hand was modeled as a cone with realistic mass and moment of inertia. The cone was connected to a stationary forearm model via a joint with two degrees-of- freedom. The five muscles that operate the wrist (Extensor Carpi Radialis Brevis, Extensor Carpi Radialis Longus, Extensor Carpi Ulnaris, Flexor Carpi Radialis and Flexor Carpi Ulnaris) were approximated as four identical muscle models (Virtual Muscle; Cheng, Brown et al. 2000) with parameters derived from morphometric and 136 physiological data. The muscles were attached to the same point on the stationary forearm but were attached symmetrically, 90 0 apart from each other to the base of the cone at their distal ends. The initial posture of the model was such that the apex of the cone pointed downwards to the ground with the stationary forearm aligned with the vertical axis as shown in Figure 1. The muscles actuated the cone about the x and z axis analogous to extension-flexion and radial-ulnar deviation rotations of the hand, respectively. The rotations of the cone were limited to 75 0 using a viscoelastic stop. The variation in moment arm during the range of all movements was tuned to be within a reasonable approximation to the parameters observed in a realistic human hand. Previously developed mathematical models of muscle spindles (Mileusnic, Brown et al. 2006) and Golgi tendon (Crago, Houk et al. 1982; Mileusnic and Loeb 2009) were used for proprioceptive feedback. We used only the primary output (Ia) of the spindles in our simulations, which provides both position and velocity feedback depending on separately modulated static and dynamic fusimotor control. The spindle secondary afferents provide essentially pure muscle length information, but relatively little is known of their interneuronal circuitry in the spinal cord. The output of the closed loop simulations was visualized in MusculoSkeletal Modeling Software (MSMS; Davoodi, Brown et al. 2003). 137 Figure 7-1: Simulation environment Biomechanical model- The human hand was approximated as a cone with realistic parameters (table). It was attached to a stationary forearm via a two degree of freedom joint. Four realistic muscle models (Virtual MuscleTM, Cheng et al 2000) actuated the cone to produce force or motion in the extension-flexion direction (in plane of drawing) and radial-ulnar direction (perpendicular to plane of drawing). Proprioceptor models- A muscle spindle model (Mileusnic et al 2006) and Golgi tendon organ model (Crago et al 1982, Mileusnic and Loeb 2009) attached to each muscle, providing proprioceptive feedback to the Spinal-Like Regulator (SLR). Control inputs- The SLR integrated proprioceptive feedback and descending commands to provide motoneuronal excitation to the muscles. These projections were associated with 184 control inputs that were differentiated as SET and GO inputs (see also Figure 2). The SET inputs continuously regulated the background activity in the spinal cord and the GO inputs produced the transition to a new state. Unlike the simple elbow joint that is used in many studies, the wrist joint presents additional challenges to the control system. The extra degree of freedom obviously adds to the mechanical complexity but even more distinctively the 138 muscles controlling the wrist movement must switch their functional relationships based on the type of task performed. For example, during wrist extension the extensor muscles function as agonists and the flexor muscles function as antagonists, but during radial/ulnar deviation, the extensor muscles (as well as the flexor muscles) oppose each other. The spinal cord circuitry is intimately related to these functional relationships between muscles, so these dynamic relationships required a new classification and corresponding synapses/connections in our spinal cord model. In our terminology we decided to call the adjacent muscles ‘partial- synergists’ because they switch from agonist to antagonist based on the type of task. The diagonal muscles that were farthest apart from each other and that always opposed each other (for example, Extensor Carpi Radialis and Flexor Carpi Ulnaris) are designated as ‘true-antagonists’ and include only the classical reciprocal antagonist circuitry between them (Figure 2). We modeled both synergistic and antagonistic circuits between the partial-synergists (Figure 3) and let the control algorithm adjust the gains of each independently, thereby establishing functional synergist and antagonist relationships as required for each task. 139 Figure 7-2: Spinal circuitry between true-antagonists The diagonal muscles are called true-antagonists because they oppose each other in all tasks. There are two such circuits in the integrated model. Five inteneuronal pathways were modeled: 1) PN - propriospinal, 2) monosynaptic Ia- excitatory, 3) reciprocal Ia-inhibitory, 4) R - Renshaw inhibitory and 5) Ib-inhibitory pathway. The inputs are differentiated as SET and GO. The SET inputs regulate the background activity in the spinal cord and consist of the gains in the neural pathways plus Ia presynaptic inhibition (small white circles marked S in the figure), and biases to the interneurons and motoneurons and fusimotor drive (gamma static and gamma dynamic) to the muscle spindles (the large white circles marked SET). 140 Figure 7-3: Spinal circuitry between partial-synergists The adjacent muscles are called partial-synergists because they switch form synergists to antagonists based on the task performed. There are four such circuits in the integrated model, each consisting of five interneuronal pathways and their SET and GO controls (same abbreviations as Figure 2). The complete SLR model has 184 control inputs. We differentiated the inputs as ‘SET’ and ‘GO’ (Figure 1), according to the terminology of Eliasmith and Anderson (2003) . The ‘SET’ inputs regulated the background activity in the spinal cord throughout the simulation and consisted of the gains of neural pathways, presynaptic inhibition, the subthreshold biases of the interneurons and the fusimotor drive (gamma static and gamma dynamic) to muscle spindles. The ‘GO’ inputs initiated movement or other state changes and maintained the new state 141 until the end of the modeled behavior. In order to explore the intrinsic properties of the spinal cord model and to test our hypothesis, we used the simplest descending inputs, namely step functions, as ‘GO’ inputs. All simulations had a duration of three seconds and the background activity in the spinal cord was responsible for maintaining a steady posture of the hand for a specified initial time (1s) before the impending ‘GO’ input. The SET gains alone mediated the rapid reflexive response of the hand during perturbation in the passive postural stabilization task. Spinal circuitry model Modeling the neuron is often a tradeoff between biological realism and computational feasibility, especially when large numbers of neurons are involved such as in the spinal cord model. The computational complexity of the model is one of the most critical factors in our simulations because hundreds of runs must be performed while the optimization algorithm converges to a desired solution. We modeled the input-output behavior of individual pools of interneurons and motoneurons using a sigmoidal transfer function (equation (1)), (x) is the input to the neuron, (a) and (b) control the slope and range of the sigmoid. The control inputs to interneurons were further differentiated as: 1) bias input that controlled the background activity in the interneuron (SET) and 2) supraspinal descending input that initiated movement (GO). The dynamic range of all neural activity was normalized between 0 and 1 and all inputs were constrained to the range ±1. The non-linearity in the sigmoidal model enabled the “reflex-gating” described when biological systems switch between behavioral states. “gains” as the input settings of these nonlinear operators in the regulator. We extracted the structure and propert cord from the literature modeled five classical types of interneurons and the between the afferents from a given (homonymous), synergist (heteronymous), and antagonist motoneuron pools ( Figs. 2, 3) : Propriospinal pathway: spindle primary Ia and Golgi tendon organs Ib) excited the priopriospinal pathways (PN in Figs. 2, 3) Meunier et al. 1991 models directly excited the homonymous motoneurons and also formed the following connections with other motoneurons: “selective” synapses whose gain could be excitatory or inhibitory for the partial synapses for the true Monosynaptic Ia excitation pathway: excited the motoneurons monosynaptically Deseilligny 1988 systems switch between behavioral states. Throughout this article, we refer to “gains” as the input settings of these nonlinear operators in the regulator. (1) We extracted the structure and properties of the known pathways in the spinal cord from the literature (summarized in Pierrot-Deseilligny and Burke 2005 modeled five classical types of interneurons and the connectivity that they establish between the afferents from a given muscle and the motoneurons of the same (homonymous), synergist (heteronymous), and antagonist motoneuron pools ( Propriospinal pathway: The proprioceptive sensory outputs (muscle spindle primary Ia and Golgi tendon organs Ib) excited the priopriospinal pathways (PN in Figs. 2, 3) (Malmgren and Pierrot-Deseilligny 1988 Meunier et al. 1991; Burke, Gracies et al. 1992). The propriospinal circuit models directly excited the homonymous motoneurons and also formed the following connections with other motoneurons: “selective” synapses whose gain could be excitatory or inhibitory for the partial-synergists and inhibitory synapses for the true-antagonist muscles. Monosynaptic Ia excitation pathway: Muscle spindle primary afferents excited the motoneurons monosynaptically (Malmgren and Pierrot Deseilligny 1988). The afferents also excited the heteronymous motoneurons 142 Throughout this article, we refer to “gains” as the input settings of these nonlinear operators in the regulator. ies of the known pathways in the spinal Deseilligny and Burke 2005) and connectivity that they establish muscle and the motoneurons of the same (homonymous), synergist (heteronymous), and antagonist motoneuron pools (α in The proprioceptive sensory outputs (muscle spindle primary Ia and Golgi tendon organs Ib) excited the priopriospinal Deseilligny 1988; Gracies, . The propriospinal circuit models directly excited the homonymous motoneurons and also formed the following connections with other motoneurons: “selective” synapses whose synergists and inhibitory Muscle spindle primary afferents Malmgren and Pierrot- The afferents also excited the heteronymous motoneurons 143 of partial-synergist muscles (Lloyd 1946; Meunier, Pierrot-Deseilligny et al. 1993). Reciprocal Ia inhibition pathway: The Ia inhibitory interneuron received monosynaptic input from Ia afferents and provided inhibitory projections to true-antagonist and partial-synergist motoneurons (Tanaka 1974; Pierrot- Deseilligny, Morin et al. 1981; Crone, Hultborn et al. 1987), as well as disynaptic recurrent inhibition (disinhibition of antagonists via the Renshaw pathway) from homonymous motoneurons (Hultborn, Jankowska et al. 1971). The Ia interneurons were themselves inhibited by the corresponding true-antagonist and partial-synergist Ia interneurons (Hultborn, Illert et al. 1976). Renshaw inhibition pathway: Renshaw interneurons (R in Figs. 2, 3) formed recurrent connections with homonymous motoneurons. They were also excited by motoneurons of partial-synergist muscles (Eccles, Fatt et al. 1954; Eccles, Eccles et al. 1961). They inhibited both partial-synergist motoneurons and true-antagonist Renshaw interneurons (Hultborn, Pierrot- Deseilligny et al. 1979; Ryall 1981). Ib inhibitory pathway: The Ib interneuron received excitatory input from Golgi tendon organs. The dominant effects of the Ib interneuronal circuitry were inhibition of homonymous motoneurons, excitation of true-antagonist motoneurons and either excitation or inhibition of partial-synergist motoneurons (Eccles, Eccles et al. 1957). 144 We excluded other known interneurons and connections listed below to limit the complexity of the model and to minimize the number of arbitrary decisions required to model poorly characterized pathways. The most detailed studies of wrist sensorimotor circuitry are in the cat, but comparisons with monkeys and humans suggest substantial refinements related to the substantially changed mechanics and behavioral repertoire of limbs used for manipulation rather than locomotion (reviewed by Illert and Kummel 1999). Because the SLR model proved to be surprisingly tractable computationally, it would be feasible to add other circuits in the future, but this would be useful only to account for behaviors that the model cannot perform well. As described below, the SLR as modeled accounted well for all motor tasks attempted so far. • All cutaneous input was omitted because the tasks to be modeled can be performed without it and because the details of these oligosynaptic circuits (e.g. Hongo, Kitazawa et al. 1989a, b) are poorly described in man. • The excitatory and inhibitory group II interneurons as well as all group II projections from the muscle spindles were omitted because their connectivity is not well-characterized (Lundberg, Malmgren et al. 1987; Jankowska 1992). Furthermore, the excitatory and inhibitory group II pathways parallel the Ia pathways to a great extent, suggesting a functional similarity between them (Lundberg, Malmgren et al. 1987). The variable gamma-static and gamma-dynamic gains in our model allow for the 145 adjustment of the relative sensitivity of Ia fiber activity to muscle length and velocity, so it captures to some extent the group II contribution to movement. • The known but poorly defined pattern of Ia projections to Ib interneurons (Jankowska, 1992) was omitted. An important property of this convergence is that it provides a reflexive adjustment of the firing sensitivity of the Ib interneuron with respect to the phase of the movement. This may be especially important for simulating more complicated tasks and we plan to incorporate some of these projections in future models. • The disynaptic inhibition of propriospinal interneurons (Illert, Lundberg et al. 1975) was omitted. The apparently diffuse connectivity of these cutaneous and proprioceptive inhibitory effects (Alstermark, Lundberg et al. 1984) suggests that they may be responsible for setting the bias of the propriospinal interneurons (SET input in our model) while the more specific excitatory projections (Illert, Lundberg et al. 1978; Malmgren and Pierrot- Deseilligny 1988) seem more likely to mediate functional relationships among the muscles and their modulation during the dynamic phase of the movement. Furthermore, the feedback inhibitory interneurons are more sensitive to input from cutaneous fibers (Malmgren and Pierrot-Deseilligny 1988b; Alstermark, Isa et al. 2007), which could be especially important for tasks that include state transitions such as between reach and grasp, which we have not modeled. 146 • Commissural interneurons were excluded from our model principally because the details of their connectivity are largely unknown and they appear not to be especially important for the tasks that we are simulating. Their bilateral projections in the spinal cord, mostly to other segmental interneurons, suggest that they may be primarily responsible for coordinating motion of contralateral limbs such as in locomotion (Jankowska 2008). All the supraspinal descending inputs projected only to the interneurons; the alpha motoneurons received no direct descending commands but the integrative outputs of the interneurons converged at the alpha motoneurons (Figures 2,3). This is a key distinction compared to previous models of spinal circuitry (Feldman, 1966; Bullock, 1992, 1993; Van Heijst et al., 1998; Bashor, 1998; Ostry and Feldman, 2003; Maier et al., 2005; Lan et al., 2005) but it is more consistent with the actual neuro- anatomy. This also appears to be the first model of spinal cord circuitry to deal with the shifting patterns of synergistic and antagonistic muscle activity that necessarily accompany multimuscle, multi-degree-of-freedom systems. Optimized control of the spinal cord model We simulated four behavioral tasks for which human performance data were available for comparison (literature references and rationales are provided with the Results): 147 • Brief perturbing torque pulses (100N x 10ms) in various directions to the hand in the initial, neutral posture • Rapid voluntary movement to a target posture (35° wrist extension) with a perturbing torque pulse • Rapid development of isometric force to a target level (either 60N extension as a sustained step or 70N extension as a transient pulse) • Adaptation to viscous curl field (rapid voluntary wrist extension performed in the presence of perturbing radial torque proportional to instantaneous velocity in extension). We anticipated that it would be difficult to use random optimization to solve such a complex, high-dimensional system, so initially we tried setting various gains “by hand” according to our intuition as neurophysiologists and control engineers. We found that it was surprisingly easy to reproduce each of the experimental behaviors by adjusting a small subset of descending controls (see examples in Figure 4 described below). Furthermore, emergent details of stability and muscle recruitment were quite realistic, even though we made little or no attempt to optimize the values of the many other control signals and we did not modulate any of the control signals. In fact, these behaviors appear to be robust emergent properties of the complete set of spinal circuitry as modeled. We then optimized all 184 control inputs (gains) using a custom gradient descent algorithm based on a simple quadratic cost function (equation 2) that penalized any deviation from the desired trajectory or state: 148 —™š˜ = ›|kšœ<k� š˜”˜k − i—˜ž”Ÿ š˜”˜k c (2) The steps in the gradient descent algorithm: 1. Random values were assigned to the control inputs initially (Monte Carlo method). However, we avoided extreme values (less than -0.5 and greater than 0.5) in order to prevent instability in the system. 2. A single control input was picked at random from the pool and optimized individually. 3. Two simulations were required to optimize each control input. The first simulation perturbed the starting control input in the positive direction and the second perturbed it in the negative direction. The value of the control input that produced the lowest cost was retained. 4. We used an “annealing curve” strategy when perturbing the control inputs, by starting with a higher value initially (Δ = ±0.2) and then reducing it for later iterations (usually by successive factors of 2 but see Discussion). 5. One iteration of the model was said to be completed when all control inputs in the model had been optimized once. 6. The whole model was then iterated multiple times until the overall cost converged to a low value. A cocontraction index was computed for all the tasks using equation (3), in which cocontraction is quantified as the product of the activation levels in the two pairs of true-antagonist muscles. This was used only to compare converged solutions, not as a component of the cost-function used during optimization. 149 Cocontraction index = ›[R‘D AY>FA × =• AY>FA S + R‘• AY>FA × =D AY>FA S] × �˜ (3) Results Stabilizing response to external force perturbation The nervous system has multiple mechanisms to counter external perturbations to the limb. The impedance of the musculoskeletal system itself created by inertial properties of the limb (force-acceleration) plus spring-like (force-length) and viscous-like (force-velocity) properties of the active muscles produces zero-delay resistive forces against sudden perturbations called “preflexes” (Loeb, Brown et al. 1999; Brown and Loeb 2000). High levels of muscle coactivation are energetically inefficient and likely to interfere with rapid movement, so short latency “reflexes” generated by proprioceptive sensors and transmitted through the circuits in the spinal cord also play a major role in resisting unexpected perturbations. Both “preflexes” and “reflexes” are under the control of the nervous system, which can effectively pre-program them to deal with expected perturbations. The “preflexes” depend on the activation level of the muscles and “reflexes” are largely governed by the background activity in the spinal cord and the levels of gamma-static (bias control) and gamma-dynamic (viscosity control) inputs to the muscle spindles. We applied an external force pulse of magnitude 100N and duration 10ms along the direction of wrist flexion to the isolated biomechanical model detached from the spinal circuitry. The passive model displayed damped oscillations about the wrist 150 joint typical of a spring-mass-damper system (dashed line in Figure 4a). The time- period of oscillation increased when gravity was removed from the system, indicating that gravity was a weak stabilizing force. Figure 7-4: Stabilizing response to external force perturbation (a) Rotation of the hand from rest position (dotted black line) about flexion-extension direction in response to a force pulse perturbation in the flexion direction (magnitude100N, duration 10ms) at 1s simulation time. The passive musculoskeletal model alone produced damped oscillations in response to the perturbation (dashed line). With the hand-tuned SLR model attached, the hand stabilized back to the resting position in 600ms (solid line). (b) Output from the Extensor (red) and Flexor (blue, plotted downward) motoneurons. (c) Corresponding extensor (red) and flexor (blue) muscle force modulation. (d) Response to perturbation after optimizing the control inputs (SET inputs) using gradient descent algorithm (solid black line). The hand stabilized back to the resting position in 200ms. Rapid voluntary movement to a position target(e) Rotation of the hand to 35 0 extension and response to the same force pulse perturbation in the extended position with minimal intuitive adjustments of the gains and step inputs to the propriospinal interneurons. (f) Corresponding extensor (red) and flexor (blue) muscle force modulation. (g) Trajectory (solid black line) after optimizing the control inputs (SET and GO) to the desired trajectory (dotted black line) using gradient descent algorithm. (h) Corresponding extensor (red) and flexor (blue) motoneuron outputs. 151 The magnitude and efficacy of preflexes depends greatly on the level of background activation (sometimes referred to as “stiffness” or “tone”) in the musculoskeletal system. We applied the same external force perturbation after attaching the spinal circuitry and feedback sensors to the plant. The hand stabilized back to the resting position in about 600ms (solid line in Figure 4a), demonstrating the stabilizing property of the spinal cord. The gains in the spinal cord were mostly left at a low value and required only minimal intuitive adjustment to produce a physiological response based on reciprocal muscle activation (Figure 4b, c). The model continued to provide a good response even after the removal of gravity from the system. When the gradient descent algorithm was used to adjust the background activity in the spinal cord (by optimizing the SET commands), it converged rapidly and stabilized the hand in 200ms (Figure 4d). Importantly, the solutions produced by our model did not rely excessively on cocontraction (see Fig. 9c below), but rather agreed well with experiments in which the opposing wrist muscles were anaesthetized to prevent any cocontraction (Hore et al., 1990). Rapid voluntary movement to a position target Cortico-spinal trajectory generation for voluntary reaching movements is assumed to shift from control of slow movements via visual and proprioceptive feedback to control of rapid movements via a feedforward trajectory generator with superimposed dynamic error compensation (Cisek, Grossberg et al. 1998). In order 152 to identify the potential contribution of the spinal cord to fast movements, we simulated a rapid, stable wrist extension from neutral to 35 0 in our model (Figure 4e). The behavior was further tested for stability by applying a brief torque perturbation (100N, 10ms) during the hold phase of the movement (at 2.3s simulation time). It was easy to obtain physiologically realistic performance by adjusting individually the feedback gains (SET) and simple step inputs (GO) to the propriospinal interneurons (Figure 4e). When the complete set of gains was optimized using the gradient descent algorithm, performance converged rapidly and tracked the desired trajectory (dotted line) even more closely (Figure 4g). The optimized solution also displayed a faster response to perturbation (Figure 4g). Voluntary isometric force to a target level While the first burst in the agonist muscle is present under both isometric and anisometric conditions, the second burst in the antagonist and the third again in the agonist (so-called triphasic burst pattern) were reported to be missing under isometric conditions in early experiments on cats (Ghez and Martin 1982). A few years later the same laboratory (Ghez and Gordon 1987) studied the role of opposing muscles in the production of isometric force trajectories in human subjects and found that for brief force rise times (<120 ms), reciprocal activation of the antagonist muscle occurred consistently. In the experiment the subjects had their arm immobilized and they were asked to produce force to a specified target level. The authors interpreted this muscle activity as indicative of similar 153 preprogrammed phasic drive from the motor cortex. We replicated the experiment in our model by constraining the wrist joint using a visco-elastic restrictor with that permitted 2mm of motion at peak force, similar to what seemed likely to be present between the bones and the apparatus through skin and padding. This results in small but potentially important departures from the purely isometric assumptions underlying this interpretation. Gradient descent was used to optimize the SET and GO inputs to the force trajectory for rapid force steps (Figure 5a) and force pulses (Figure 5b). Rapid modulation of agonist activity (Figure 5c, d) could be explained by modulation of spindle afferent activity (not illustrated) as a result of stretch of their elastic tendons. Antagonist bursts were produced for rapid force modulations (Figure 5c) but were absent for slower force trajectories (Figure 5d), as was reported experimentally (Ghez and Gordon 1987). These were associated with the smaller modulations of their homonymous spindle afferents as a result of the compliance of the visco-elastic restrictor. 154 Figure 7-5: Voluntary Isometric force to a target level (a) The net reaction force (solid black line) produced by the isometric muscles after optimizing the control inputs to the desired force (dashed black line) using gradient descent algorithm. (b) Pulse force trajectory (solid black line) produced after optimizing the control inputs using gradient descent algorithm. (c) Agonist (red) and Antagonist (blue) motoneuron output for brief rise time (100ms) pulse-force trajectory. Tasks replicating brief rise time force trajectories (<120ms) consistently produced antagonistic pulses in both pulse and step force trajectory tasks. (d) Agonist (red) and Antagonist (blue) motoneuron output for long rise time (250ms) step-force trajectory task. The antagonist pulse was distinctively absent for intermediate (120-200ms) and long rise-time (>200ms) force trajectory tasks. Adaptation to viscous curl force fields The adaptation of human reaching to novel dynamic environments such as an opposing curl-field has been widely accepted as one of the most valuable testbeds for the notion of internal models and the learning process involved in their development and adaptation (see Kluzik, Diedrichsen et al. 2008 for recent examples of this extensive literature). In the experiment the subject makes reaching 155 movements with a robot that generates a perturbing force proportional to the velocity of the actual movement and perpendicular to the direction of intended movement. The trajectories are initially skewed in the direction of the perturbation but gradually straighten with adaptation. When the field is unexpectedly removed, the trajectories show mirror-image after-effects. In an attempt to find the limits of the spinal cord’s potential role in motor adaptation, we replicated the experiment in our model. We defined the intended direction of movement of the hand about the x-axis (flexion-extension) and applied viscous perturbing force about the z-axis (radial-ulnar deviation) in proportion to the velocity in the x-axis. Figure 6 illustrates the surprisingly good performance obtained using a simple gradient descent solution with unmodulated SET and GO descending commands for gains. The extension movement (Figure 6a) follows the desired trajectory almost as well as for the unopposed extension solution in Figure 4g; the initial radio-ulnar deviation of about 35° before training has been reduced to almost zero (Figure 6b). The mechanisms that underlie this performance appear to involve a combination of asymmetrical fusimotor gains (Figure 6e) and asymmetrical activation of propriospinal neurons (Figure 6g) and resulting muscle force (Figure 6h). The after-effects are also surprisingly physiological when the curl-field is removed from a system that has adapted to it (Figure 6c, d). 156 Figure 7-6: Adaptation to viscous curl force fields 157 (a) Tracking a desired trajectory (dotted black line) in the presence of viscous force field perturbations. The gradient descent algorithm optimized the control inputs to produce extension (solid black line) that closely tracked the desired trajectory. (b) Radio-Ulnar deviation in the presence of viscous curl force fields reduced from 35 0 to nearly 0 0 (solid black line) after optimization. (c,d) After-effects when the curl force field was removed. (e) Primary afferent response from the muscle spindles in each of the four muscles after adaptation (two extensor muscles in red plotted upward; two flexor muscles in blue plotted downward). (f) Responses from the four Golgi tendon organs after adaptation. (g) Output of motoneurons exciting the extensor muscles (red) and flexor muscles (blue), after adaptation. (h) Extensor (red) and flexor (blue) muscle force modulation after adaptation. In order to explore the variability of adaptation across subjects, we took advantage of the fact that the SLR generally produces many “good enough” solutions for any given task when started with random initialization gains (see Figure 7 below). We used several such solutions for unperturbed, rapid wrist extension as starting points for adapting to the curl fields. This resulted in generally adequate but substantially different performance for these functionally equivalent starting points, in terms of speed of convergence to a solution, quality of the solution, and the magnitude of after-effects. This would be consistent with the general observation that subjects differ in how quickly they learn to compensate for unusual and complex loads such as curl fields (e.g. Fig. 11 in Shadmehr and Mussa-Ivaldi 1994). 158 Figure 7-7: Analysis of gain values (a) Learning curves for 10 solutions of rapid ramp-hold movements all initialized to different random settings. Even though the initial cost values were considerably different, all curves converged rapidly to stable solutions. (b). The final values (ordinate axis) for 54 of the gains from each of two solutions (red and blue circles) from (a) that achieved very similar cost and kinematic details. They differ substantially in many details, often including both magnitude and sign. (c). Distance between the starting and final positions in the high-dimensional state-space of gains (square root of the sum of squared distances in all 200 gain axes). The 45 possible pair-wise comparisons, ordered according to their distances from each other after optimization. 159 The subjects in force-field experiments showed after-effects of force adaptation even though they had explicit knowledge that no external force would be applied to the reaching hand (Kluzik, Diedrichsen et al. 2008). This suggests that force adaptation cannot be switched on or off by cognitive cues alone but may also involve a spinal level adaptation that takes time to readjust. In the SLR, the errors produced due to the unexpected dynamic changes were transduced by the proprioceptive sensors, which in turn evoked reflexes to reduce the error. Our results suggest that the gains in the regulator circuitry can be readjusted to eliminate these trajectory deviations almost completely. We acknowledge that the model does not account for various other observations in the experiments such as the generalization of adaptation across arms that appears to be in the extrinsic coordinates of the task (Criscimagna-Hemminger, Donchin et al. 2003). There are substantial interhemispheral connections as well as uncrossed descending circuits and bilateral commissural spinal interneurons (which we have not modeled) that may contribute to such generalization, even without an explicit internal model in extrinsic coordinates. Multiple Local Minima Figure 7a shows learning curves for 10 solutions of rapid ramp-hold movements, all initialized to different random settings. Cost reflects the deviation from the desired trajectory as defined in Equation 2. Even though the initial cost values were considerably different, all converged rapidly to stable solutions. The final values 160 were distinctly different from each other but were all within typical variability of experimental performance (note log scale of cost function). We further examined the solutions themselves to see how they differed from each other. Figure 7b compares the final values (ordinate axis) for 54 of the gains from two of the solutions from Figure 7a that achieved very similar cost and kinematic details. They differ substantially in many details, often including both magnitude and sign. This suggests that the state-space consists of many, widely separated local minima that can be used to perform the task successfully. We further tested the distribution of local minima by examining the distances between the starting and final positions in the high-dimensional state-space of gains (square root of the sum of squared distances in all 184 gain axes), illustrated in the bar-graph in Figure 7c for the 45 possible pairwise comparisons, ordered according to their distances from each other after optimization. There was a weak tendency for solutions that were close to each other in space to have started closer to each other initially, consistent with the expected properties of gradient descent. There was no tendency for the normalized cost differences of the starting positions or optimized solutions to correlate with separation of the solutions in state-space, consistent with the notion that the local minima are widely distributed throughout the state-space and not greatly different in actual values or local steepness. 161 Learning Rate Across all the simulated tasks we noticed a consistent and prominent characteristic in the gradient descent process, regardless of task complexity, random starting positions or initial performance: The vast majority of the reduction in cost was obtained in the first iteration that used the biggest step change in gain. In all our simulations we used a fairly shallow “annealing curve” strategy, starting with a larger change in gain in the first iteration (Δ=±0.2) to escape from poor local minima and then reducing the change to smaller values (typically ±0.1, ±0.05, ±0.01) in subsequent iterations. Figure 8 shows one illustrative solution for each of the five tasks that we have modeled. Further optimization from this point produced only very small reductions in the cost even though there were substantial changes in the gains, indicating broad local minima. We have tried several different annealing curves (patterns of decreasing gain steps explored in successive iterations), but this finding appears to be robust, suggesting that most local minima are well-separated from nearby local minima. Clearly, under some circumstance it may be desirable to force systems out of local minima in order to find the global minimum for truly optimal performance, as opposed to “good enough” performance. That may require more complex gradient descent strategies in which more than one gain is varied at a time. This notion seems to have some subjective resonance with the strategies used by coaches to force athletes out of “bad habits” (Lyndon 1989). 162 Figure 7-8: Typical learning curve for each task All tasks started with random starting values for the control inputs, which were then optimized using gradient descent algorithm. One iteration of the model was said to be completed when all control inputs in the model had been optimized once. The common feature across all tasks was that almost all of the cost reduction was obtained in the first iteration with little improvement in subsequent iterations. Comparison with Servo-Control In order to compare the performance of our model with a sufficiently complex model with the same sensors and actuators but without the specific interneuronal types, we developed a classical servo-control model and replicated the same five tasks. The motoneurons received direct feed-forward commands plus homonymous positive and negative feedback from the muscle spindle and Golgi tendon organs, respectively (Figure 9a), similar to the “autogenetic stiffness servoregulator” proposed by Houk (1979). The only non-homonymous projection to the motoneuron was the negative feedback from the true-antagonist muscle spindle. 163 The mean performance cost produced by the new model was significantly higher in all the tasks compared to the SLR model (Figure 9b). Even for relatively good performances, servocontrol produced unphysiological levels of co-contraction in muscles (note log scale in Figure 9c), a strategy that is highly inefficient because it requires high metabolic energy. The optimized performance of the servo-controller was highly sensitive to the starting values of the gains and some solutions had large deviations from the desired behavior. The local minima in the solution space appeared to be very broad and it was often difficult to escape from poor minima even with multiple iterations. In contrast the SLR model produced performance costs that were always within observed experimental variability regardless of initialization conditions, and the mean and standard deviations of the costs across similar tasks were considerably lower. The SLR model relied heavily on proprioceptive feedback to improve performance in all the tasks with little co- contraction (Fig. 9c). The local minima in the solution space were quite robust and in all cases were adequate to produce physiologically realistic performance. 164 Figure 7-9: Servo-control model 165 (a) Circuit diagram of a classical servo-control model. The motoneurons received direct feed-forward commands plus homonymous positive feedback from the muscle spindle primary afferents and negative feedback from the Golgi tendon organs. They also received negative feedback from true- antagonist muscle spindle primary afferents. All of these commands and gains and fusimotor biases were adjusted by random gradient descent as for the SLR model. (b). Cost comparison between the servo-control model, SLR model and experimental performance (Ghez and Gordon 1987, Wierzbicka et al 1991, Liles 1985)is shown for each of the tasks we modeled. The shaded box with the dotted outline represents performance levels for those subjects who adopted muscle coactivation rather than the more typical triphasic burst pattern. (c). Cocontraction comparison similar to (b), see text. Sensitivity of optimized solutions The tendency of the learning curves to plateau rapidly at low costs regardless of annealing curve (Figs. 7, 8) suggests that the optimized solutions of the SLR model were relatively robust to changes in individual gains. A more systematic test of this was generated for a typical converged solution for the rapid voluntary extension task. After three iterations with Δ = 0.1 for all gains, the cost had plateaued to the low level indicated at Delta 0 in the center of Figure 10a. From that starting gain set, each gain was systematically varied by steps of ± 0.05, 0.1 or 0.2, resulting in a family of changes to the Cost. The introduction of the finer steps of 0.05 resulted in small improvements for some gains, as expected. Some of the gains had little effect on this task; those same gains were often more critical for other tasks (e.g. gains related to force feedback tended to be more important for the isometric force tasks). A few of the gains (thick lines with symbols; key in legend) had steeper effects on performance, often for changes in one direction but not the other. Because the starting values of these gains could vary from zero to ±1, the fractional value 166 represented by these deltas varied widely and might be more relevant to their sensitivity to neural noise. The values for Δ = ±0.05 were replotted as percentage change in Figure 10b, with infinity denoting starting gains of zero. Two of the gains were still quite sensitive in both directions, but their effects were reciprocal, so that an increase in one could be cancelled by a decrease in the other. Interestingly, the synergistic extensor muscle that must work symmetrically in this task did not exhibit a similar pattern of sensitivity. After deliberately fixing either of these most sensitive gains to a value that produced high cost, running one iteration to adjust the remaining gains was able to reduce the cost back down to the human performance range (dashed lines). This was accomplished by changing a large number of gains depending on the random order in which they were tested, not necessarily the gain exhibiting the reciprocal relationship noted above. These findings are consistent with the low costs after the initial iterations with the relatively coarse Δ = 0.1 and the broadly different gain solutions with similar performance (e.g. Figure 7b). 167 Figure 7-10: Sensitivity analysis (a) Starting with a set of converged gains after three iterations with delta = 0.1, each gain was individually adjusted up and down by Deltas of 0.05, 0.1 and 0.2 and the resulting Cost plotted on the ordinate; dashed lines indicate range of normal human performance. Traces exhibiting high sensitivity in at least one delta direction are highlighted in color (key: red down arrow = GO drive to propriospinal interneuron to EU muscle; green up arrow = SET drive to same; mauve circle = GO drive to propriospinal interneuron to FR; purple square = SET drive to same; green star = Ib feedback to FU). (b) Cost values for Δ=±0.05 replotted as percent gain change on abscissa (“inf” denotes starting value = 0). 168 Generalization of optimized solutions One important property of motor strategies learned under one set of conditions is that they generalize to other, similar conditions that were not part of the training set. We tested whether this was a property of the SLR strategies to resist brief force perturbations such as illustrated in Figure 4d. Initially we trained the SLR to resist force perturbations in the four perpendicular anatomical axes of the wrist, i.e. flexion, extension, radial and ulnar directions (red axes lines in Figure 11). The algorithm thus optimized the gains to reduce the sum of the costs calculated from perturbations in all four anatomical axes, delivered sequentially, one after the other. After training the converged solution set provided good response to perturbation in all four trained directions. We then tested the ability of the converged solution to resist similar perturbation in new untrained directions such as pulling directions of the individual muscles, i.e, rotated 45 0 from the anatomical axes (blue axes lines in Figure 11). The responses were almost equally rapid and accurate as for the trained directions (Figure 11). Thus one solution set was able to generalize and resist perturbation in any direction in the workspace. 169 Figure 7-11: Ability of SLR strategy to generalize to new task conditions (a) Response to brief force perturbation along the extension direction (black arrow) before (dotted red line) and after (solid red line) training the SLR to resist in the anatomical axes of the wrist, i.e, flexion, extension, radial and ulnar directions (red axes lines). (b) Rotation of the wrist along ext-flex & rad-uln axes in response to the same force perturbation along untrained axes (blue axes lines), the pulling direction of the ECR muscle. (c) Response of the SLR due to perturbation along the pulling direction of FCU muscle. 170 Discussion Limitations of the Model The selection and design of the modeled components entailed many omissions and simplifications from the known physiology and mechanics of this neuromusculoskeletal system. Despite these limitations, it was relatively easy for a simple gradient descent learning algorithm to achieve realistic performance in terms of both the performance goals of the task and realistic patterns of muscle activation regardless of starting conditions. Nevertheless, it is worth asking whether these simplifications themselves contributed to this finding. They include the symmetrical arrangement of the muscles, the sigmoidal input-output functions for all the neurons (omitting more complex nonlinearities such as plateau potentials; Hultborn, 2002), and the omission of certain classes of interneuronal connections as detailed in the Methods. The servo-control model shared these properties but was notably less successful. Thus, some sort of regulator-like structure may be necessary, at least when the command structure is assumed to be highly limited (e.g. SET & GO step functions as employed herein). The modeled spinal-like regulator appears to be sufficient under such circumstances, but it would be interesting to replace some modeled pathways with some excluded ones whose connectivity suggests similar functionality, and then to compare their performance. It is also possible that inclusion of some of the omitted pathways will be required to control more difficult tasks or more complex musculoskeletal systems. We are now 171 extending the SLR to control a two-joint planar arm that includes both uni- and bi- articular muscles (Tsianos et al., 2009), a well-studied experimental preparation that presents a richer set of biomechanical challenges. In the future, we hope to replace the SLR circuitry with equally complex sets of interneurons with randomized connections to understand whether the capabilities of the SLR arise from its specific patterns of connectivity as described in the literature or are simply an emergent property at a critical level of complexity. The General Utility of Regulators The spinal-like regulator (SLR) has a large number of control points, but it turned out to be surprisingly easy to find simple patterns of descending inputs that caused it to replicate actual performance details of a wide range of motor behaviors. This appears to be a consequence of the state space defined by the SLR, which has many local, “good enough” minima for each of the tasks we simulated. Importantly, regardless of starting gains, the SLR never became trapped in a local minimum that produced less than satisfactory performance. Against common intuition, we found that the larger the number of inputs that were optimized by the algorithm, the faster it converged to better performance. This is in contrast to the servo-control model, which converged more slowly and often to poor solutions. Poor performance, but not slow convergence, might reflect the substantially fewer interneurons and adjustable gains in the servo-controller compared to the SLR. 172 One obvious complexity that we have avoided concerns the temporal modulation of descending commands during tasks. We were surprised by the range of dynamic behaviors that the model could reproduce using simple SET & GO step functions, but we expect this to break down for more complex tasks. Recordings of cortical activity during even simple tasks often show substantial temporal modulation that is more complex than a step function centered on the task initiation (Churchland and Shenoy 2007). In fact, these patterns seem to have more in common with the wide variety of patterns described for spinal interneurons (Maier, Perlmutter et al. 1998; Perlmutter, Maier et al. 1998) than with the canonical parameters describing the task kinematics (e.g. direction, distance, velocity) or kinetics (e.g. individual muscle activity). Interestingly, complex EMG patterns in monkey arm muscles were surprisingly well-predicted from cortical single-unit activity using nonlinear combinatorial functions such as might be embodied in spinal interneurons (Song, Hendrickson et al. 2008) rather than more simple combinations (Westwick, Pohlmeyer et al. 2006). It is also important to consider the effects of inter-trial variability in the descending commands. Churchland et al. (2006) determined that approximately 50% of the variance in performance was already present in these cortical neural signals rather than arising from the “motor noise” that has been attributed to discrete recruitment of size-ordered motor units (Jones, Hamilton et al. 2002). It is also likely that the cortical variability is itself correlated among multiple units (Maynard, Hatsopoulos et al. 1999). Churchland et al. interpreted this variability as 173 noise, but it may be indicative of purposeful exploration of the complex solution space afforded by an SLR to which these neurons project. Such exploration might be useful to free up the limited computational capacity of cortex so that other tasks can be learned. Doyon and Benali (2005) attributed the later stages of motor consolidation to redistribution via the cortico-striatal or cortical-cerebellar systems, but it may also depend on the solution space of the SLR. Adding noise or purposeful variability to the command signals that are responsible for setting the SLR gains during iterative adjustment seems likely to prevent the solution from settling into local minima for which any individual gain has a steep cost function (see Figure 10 and related Results regarding sensitivity). Interpretation of Deafferentation Experiments Many researchers attribute a dominant role to descending commands rather than segmental feedback because studies have shown that both chronically and acutely deafferented subjects can produce some temporal details of normal muscle activation, specifically a triphasic burst pattern during rapid movements that is similar to that produced by normal subjects (Hallett, Shahani et al. 1975; Rothwell, Traub et al. 1982; Sanes and Jennings 1984; Forget and Lamarre 1987; Berardelli, Hallett et al. 1996). These results, however, do not prove that the behavior is normally achieved via open loop commands. It is possible that in the absence of sensory feedback to the regulator, the brain could have learned to produce a similar motor program, given that it is a necessary strategy of muscle recruitment for 174 performing the task successfully. Interestingly, a deafferented patient exhibited a tri-phasic burst pattern in Forget and Lamarre’s study in 1983 but was unable to produce it two years prior. Furthermore, chronically deafferented patients exhibit significantly different EMG timing and worse kinematic performance (Rothwell, Traub et al. 1982; Forget and Lamarre 1987; Gordon, Iyer et al. 1987), which could mean that the brain is intrinsically an inferior source of such rapidly modulated motor commands compared to a regulator with phasic afferent feedback. The acutely deafferented patients in Sanes and Jennings’ (1984) study were able to produce remarkably crisp EMG patterns and relatively better kinematic performance than the chronically deafferented patients of the other studies. However, the ischemic block that they imposed affected afferent activity from muscles below the elbow, leaving intact both the efference copy from their motoneurons via the Renshaw interneurons and all proprioceptive feedback from all proximal muscles. The widespread extent of such feedback circuitry (Cavallari, Katz et al. 1992) and the strong modulation of proximal muscles to stabilize posture during phasic distal movements was fully appreciated only subsequently (Massion 1992). Only a small portion of the actual feedback to a spinal regulator would have been removed as a result of the ischemic block. This might account for the relatively minor effects on EMG amplitudes and kinematic trajectories that were actually observed. For the multiarticular model of elbow and shoulder muscles now under development (Tsianos et al., 2009), it should be possible to simulate such partial deafferentation on a converged control scheme for an SLR. 175 Deafferentation and ablation experiments provide information about what structures are necessary to perform a task only when the task fails; they never provide information about what structures are sufficient. Conversely, models can demonstrate whether a mechanism is sufficient to perform a task when they succeed, but will never prove what is necessary. Our general hypothesis is that the spinal cord (together with other regulatory circuits with similar connectivity; see below) plays a major role that facilitates learning and performing voluntary tasks. Models cannot prove that this is the case, but they can demonstrate the extent to which this hypothesis is consistent with observed behavior and should be considered when interpreting neural activity during such behavior. Hierarchical Architecture The random gradient descent algorithm appears to be at least qualitatively similar to the manner in which the infantile brain learns to use the spinal cord or individual subjects acquire and refine any new motor skill. The important question is where do the signals being adjusted during this learning process actually go. If those signals project to something like our SLR model, that entity must substantially define and may generally simplify the learning of motor tasks by the brain. Neurophysiologists recording neural activity in motor cortex have looked for correlations with task kinematics or kinetics and inevitably find them, but this sheds no light on the long-standing debate over which parameters of motor behavior are 176 commanded and encoded by the motor cortex (Fetz 1992; Loeb, Brown et al. 1996; Scott 2000; Churchland and Shenoy 2007). Previous formulations of hierarchical control (Loeb et al., 1999) have associated the regulator with the spinal cord alone, leaving the motor cortex to function only as a source of the feed-forward commands as described herein. In fact, it is more likely that many, if not all, centers in the sensorimotor system integrate sensory feedback from lower centers with command input from higher centers, accounting for the observed temporal complexity at all levels. Furthermore, some of the functionality attributed to the SLR in the present model may actually occur in similar circuitry residing in other structures in the central nervous system (e.g. rubromotoneuronal system included in premotoneuronal model by Maier et al., 2005). Experiments that apply brief perturbations while recording from single units (e.g. Weber and He 2004) can help to identify such functionality, but they need to be interpreted in the light of a more general theory of computation. A general notion of “regulation” is illustrated in Figure 12, in which a regulator is any structure whose outputs depend on multiple sources of feedback from the lower level according to programmable gains (control signals) set by a higher level. This concept extends down the hierarchy to account for muscle physiology and musculoskeletal mechanics. For example, the force output of muscles is highly dependent on kinematic conditions imposed by the motion of the limb according to the state of activation of the muscles (preflexes as defined by Brown and Loeb, 2000); similarly, the interaction of limbs with objects is highly dependent on the 177 mechanical impedance of the limb according to its posture and stiffness as set by muscles (Hogan, Bizzi et al. 1987). For simple systems that are invertible or locally linearizable, analytical methods can be used to compute optimal solutions for regulator settings (He, Levine et al. 1991; Todorov and Jordan 2002). However, the brain is unlikely to employ such methods, making it important to demonstrate that realistic systems are amenable to trial-and-error learning as demonstrated herein. Figure 7-12: Generalizing the concept of hierarchical control Each level of the hierarchy from cerebral cortex (Cx) to spinal cord to musculoskeletal plant performs a regulatory function for the level above and acts as a controller for the level below. The label inside the each feedback loop indicates a typical manifestation of the function at each level. See Discussion. 178 Industrial robots are meticulously programmed to perform desired tasks and then carefully insulated from the vicissitudes of human operators and changing circumstances. In contrast, the brain delights in exploring new ways to perform old tasks, secure in the knowledge that the emergent behavior will be at least acceptable and stable. Muscles, proprioceptors and interneurons are complex but such biological systems have properties that appear to simplify the tasks of learning and producing complex movements. That alone would provide a basis for their evolution and retention. The functional connectivity that justifies extending this notion up the hierarchy from the spinal cord (where it was originally formulated) to the rest of the brain has long been known to exist (e.g. proprioceptive fields and transcortical reflexes; Lee and Tatton 1975; Evarts and Tanji 1976) but it has yet to be described in sufficient detail to permit the type of modeling presented here. Acknowledgments We thank R. Davoodi, M. Kurse, J Weisz and D. Song for their technical assistance and insights and C. Ghez and E. Fetz for valuable discussions. This work was supported by the NSF Engineering Research Center for Biomimetic MicroElectronic Systems. G. Tsianos is supported by a Myronis Foundation Fellowship. 179 Chapter 8: Modeling the Potentiality of Spinal-Like Circuitry for Stabilization of a Planar Arm System George A. Tsianos, Giby Raphael, and Gerald E. Loeb © 2011 Elsevier. Reprinted with permission from: Tsianos, G. A., Raphael, G., & Loeb, G. E. (2011). Modeling the potentiality of spinal-like circuitry for stabilization of a planar arm system. Prog Brain Res, 194, 203-213. Preface The following article, published in Progress in Brain Research, describes the application of the SLR control scheme to a shoulder-elbow system whose motion is constrained in the horizontal plane. This system was chosen to investigate the lower motor system’s role in resolving complexities such as redundant musculature, intersegmental dynamics, and high dimensional control space (438 gains to be set). It successfully learned to resist a brief, oblique perturbation applied to the endpoint given the systematic use of cocontraction during the learning process. Adding a modest level of cocontraction during the initial phase of learning and subsequently removing it facilitated the search for solutions whose performance was within subject variability. 180 Author contributions George Tsianos and Gerald Loeb wrote the manuscript, designed the SLR for the arm, developed modeling experiments, and analyzed results. Giby Raphael assisted with the design of the SLR. Abstract The design of control systems for limb prostheses seems likely to benefit from an understanding of how sensorimotor integration is achieved in the intact system. Traditional BMIs guess what movement parameters are encoded by brain activity and then decode them to drive prostheses directly. Modeling the known structure and emergent properties of the biological decoder itself is likely to be more effective in bridging from normal brain activity to functionally useful limb movement. In this study, we have extended a model of spinal circuitry (termed SLR for spinal-like regulator (see Raphael, Tsianos et al. 2010) to a planar elbow-shoulder system to investigate how the spinal cord contributes to the control of a musculoskeletal system with redundant and multiarticular musculature and interaction (Coriolis) torques, which are common control problems for multi-segment linkages throughout the body. The SLR consists of a realistic set of interneuronal pathways (monosynaptic Ia-excitatory, reciprocal Ia-inhibitory, Renshaw inhibitory, Ib- inhibitory and propriospinal) that are driven by unmodulated step commands with learned amplitudes. We simulated the response of a planar arm to a brief, oblique impulse at the hand and investigated the role of cocontraction in learning to resist it. 181 Training the SLR without cocontraction led to generally poor performance that was significantly worse than training with cocontraction. Furthermore, removing cocontraction from the converged solutions and retraining the system achieved better performance than the SLR responses without cocontraction. Cocontraction appears to reshape the solution space, virtually eliminating the probability of entrapment in poor local minima. The local minima that are entered during learning with cocontraction are favorable starting points for learning to perform the task when cocontraction is abruptly removed. Given the control system’s ability to learn effectively and rapidly, we hypothesize that it will generalize more readily to the wider range of tasks that subjects must learn to perform, as opposed to BMIs mapped to outputs of the musculoskeletal system. Introduction Moving limbs in a purposeful manner, whether it is a simple reaching movement or fine manipulation of an object, is an elaborate process that requires sophisticated integration of volitional commands and sensory feedback. The design of control systems for sensorimotor prostheses seems likely to benefit from an understanding of how such control is achieved in the intact system. Many regions of the brain participate in this process, but all of their output signals are integrated in the spinal cord. The spinal circuitry consists of a variety of reasonably well-characterized interneurons that are highly evolved and conserved throughout mammalian and even vertebrate evolution (Pierrot-Deseilligny and Burke 2005). In particular, the 182 large majority of corticospinal neurons have few or no direct projections to spinal motoneurons (Rathelot and Strick 2009), projecting instead to spinal interneurons where their signals are integrated with various somatosensory afferents and recurrent motoneuron signals. Depending on one’s perspective, any given muscle recruitment can be described either as the result of a descending command that has been modulated by segmental feedback or a segmental reflex whose gain has been set by descending commands; given the circuitry, they are functionally indistiguishable. A brain-machine interface naturally focuses on the cortical command signals. Cortical activity recorded from non-human primates during trained motor behaviors can be correlated with experimental measures of the kinematics or kinetics of the performance. The decoded cortical signals can be used to recreate those kinematics or kinetics in a robotic or virtual simulation of the limb (see other chapters in this volume). Because of the mechanical coupling within the musculoskeletal system, however, similar correlations can be obtained with a wide variety of measures of the task (Churchland and Shenoy 2007). Furthermore, the input-output relationships shift substantially with small changes to the task (e.g. changes in limb posture unrelated to the end-point trajectory being controlled(Scott and Kalaska 1995). This suggests that the dimensionality of neural activity is significantly larger than the set of movement parameters that are hypothesized to be encoded (see Churchland and Shenoy 2007). This also indicates that the correlations do not reflect the coordinate frame in which the brain normally 183 computes command signals. Furthermore, these correlations may actually reflect cortical inputs from higher motor planning centers (e.g. parietal cortex) and/or somatosensory feedback and efference copy signals from lower centers such as the spinal cord. Rather than guessing what coordinates the brain might use and building decoders based on correlations observed, we can start with the known structure of the biological decoder itself. Fortunately a fair amount of spinal connectivity is known (see Jankowska 1992; Pierrot-Deseilligny and Burke 2005). Modeling of musculoskeletal systems is sufficiently advanced to support the development of realistic model systems in which the potentiality of the components can be appreciated. Modeling tools can be used to gain insight into the spinal cord’s contribution to various behavioral phenomena such as kinematic performance, stability, energy consumption and learning. Alternatively, they can also give us insight into the extent to which these aspects need to be specified explicitly by higher centers or treated as emergent properties of the system being controlled. In previous research, we obtained surprising results from a realistic model of the spinal circuitry operating a model of a two degree-of-freedom wrist with four muscles (Raphael, Tsianos et al. 2010). The model consisted of a realistic set of interneurons whose descending commands were simple step functions with learned amplitudes. The model was called a “spinal-like regulator” (SLR) because it included elements that may actually be located in supraspinal circuits and excluded some known spinal interneurons whose connectivity or roles were less well- 184 characterized. Despite having an oversimplified brain whose outputs were limited to unmodulated steps, a simple gradient descent algorithm rapidly discovered physiological solutions for a wide range of tasks. Even with a large number of control inputs, the simple learning algorithm always converged rapidly to solutions similar to published normal behavior regardless of the random starting point of the search. This is surprising because the large number of control inputs would theoretically create a complex solution space with many undesirable local minima. The fact that training always resulted in good performance implies that the structure of the spinal circuitry facilitates learning by crafting a solution space consisting of many local minima that are good enough for many common tasks. Furthermore, details of muscle activity during the learned behaviors appeared to be physiological (e.g. minimal cocontraction) even though muscle activation was not included in the training criteria. This suggested that the structure of the spinal cord is predisposed toward metabolically efficient behavior. In this study, we have extended this modeling scheme to a planar elbow- shoulder system to investigate how the spinal cord contributes to the control of a musculoskeletal system with redundant and multiarticular musculature and interaction (Coriolis) torques, which are common control problems for multi- segment linkages throughout the body. 185 Methods Simulation environment The neuro-musculoskeletal system shown schematically in Figure 1 includes realistic models of muscles, proprioceptors and spinal circuitry in conjunction with a simplified model of the brain. Models of individual components have been described in other publications and are summarized here. Figure 8-1: Schematic overview of the neuromusculoskeletal system of the planar arm Descending commands from the brain model and proprioceptive feedback from muscle spindle and Golgi tendon organ models project to interneurons in the spinal-like regulator (SLR). The interneurons integrate this information and send it to the alpha motoneurons that drive the muscles. The brain model also delivers fusimotor input to the muscle spindles, effectively setting their transduction sensitivity. 186 Musculoskeletal model The musculoskeletal system represents an arm whose motion is constrained within the horizontal plane (Brown and Loeb 2000). The 3-segment skeleton is made up of torso, upper and lower arm segments that are linked by hinge-like shoulder and elbow joints. Each joint is operated by a pair of antagonist muscles that provide flexion and extension torques. In addition, a pair of biarticular muscles provides flexion and extension torques across both joints. See Figure 2 for a detailed description of the musculoskeletal parameters. Figure 8-2: Musculoskeletal system parameters The proximal and distal arms have identical dimensions and mass, which is uniformly distributed over each segment. In the posture shown in the figure, the muscles are arranged symmetrically about the proximal segment. The monoarticular muscles have one attachment point at the center of the proximal segment and another one 3cm away from the joint they actuate. The biarticular muscles attach 3cm away from the elbow and shoulder joint on the same side of the proximal segment. 187 The muscle model used in this study (Tsianos, Rustin et al. 2012) is a modified version of Virtual Muscle (VM) presented in Cheng et al. 2000 (Cheng, Brown et al. 2000). The new muscle model is more computationally efficient and computes energy consumption in addition to force over a wide range of stimulation conditions. It accurately captures the nonlinear effects of firing rate, kinematics, and fiber composition on force production and energy consumption. Tendon plus aponeurosis are modeled as a nonlinear elastic component in series with the contractile machinery. Under dynamic conditions, such series elasticity results in substantial differences between the kinematics of the whole muscle and of the muscle fascicles and spindles, which has significant effects on force production, energy consumption and proprioceptor activity. Proprioceptor models Each muscle in our system includes models of muscle spindles (Mileusnic, Brown et al. 2006) and Golgi tendon organs (GTO; Mileusnic and Loeb 2009). The muscle spindle model generates a response depending on fascicle kinematics and fusimotor excitation, with separate gamma static and gamma dynamic control of length and velocity sensitivity, respectively. Although the model captures both group Ia and II afferent responses, we used only the Ia response in our system because we omitted spinal circuitry associated with group II feedback (see Raphael, 188 Tsianos et al. 2010). The GTO model generates a response that represents activity from an ensemble of group Ib afferents in response to whole muscle tension. Spinal cord model The spinal cord model is composed of classical interneuronal circuitry described in the experimental literature plus fusimotor control for the muscle spindles. It includes the following pathways: monosynaptic Ia-excitatory, reciprocal Ia- inhibitory, Renshaw inhibitory, Ib-inhibitory and propriospinal interneuronal pathways. These circuits between a given pair of muscles are largely defined by their functional relationship, which can be synergist, antagonist, or variable depending on task, which we term partial synergists. The connectivity of these relationships for each type of interneuronal circuit is described in detail in Raphael et. al. 2010. Given that the building blocks of the overall spinal network have already been defined, the major challenge in this study was to determine the functional relationships among the arm muscles to construct the network specific to this system. Although monoarticular muscles crossing a single joint are obviously antagonistic, the interaction torques among joints in the arm makes it difficult to intuit the underlying muscle activity, hence whether and when a given pair of muscles act as synergists or antagonists. Furthermore, the system is kinetically redundant (meaning that multiple sets of muscle activation patterns can accomplish 189 the same movement), which further complicates the relationships among the various muscles. Kinetics and EMG studies of planar arm movement provide descriptions of muscle coordination patterns associated with reaching tasks, thus giving us insight into these functional relationships. We used the active torque analysis presented in Graham et. al. 2003 to identify the functional relationships among monoarticular muscles crossing different joints. The direction of active joint torques agreed with monoarticular muscle activity from several movements presented in Hasan and Karst 1989, therefore, active torque direction was a good indicator of which muscle was being recruited. We found that although reaching movements typically require that the shoulder and elbow rotate in opposite directions, the direction of active torques was often the same. In fact, all combinations of active joint torque direction between the two joints were observed, suggesting that any given pair of muscles that cross different joints act as partial synergists. EMG data from Hasan and Karst 1989 were also used to gain insight into the relationships between these muscles and the biarticular muscles, whose individual contributions cannot be deduced with confidence from net joint torques. We found that biarticular muscles could be recruited in or out of phase with the monoarticular muscles that had the same actions at the joints they crossed, suggesting that they have partial synergist relationships with the rest of the set. In summary, each monarticular muscle is modeled as antagonist to the monoarticular muscle crossing the same joint and partial synergist to both 190 monoarticular muscles crossing the other joint. Each biarticular muscle is modeled as a partial synergist to all muscles in the set, including each other. The overall network consists of 340 local projections (e.g. afferent and interneuronal pathways) whose activity is modulated by the brain (see next section). These pathways project among 24 interneurons (four classical types for each of the six muscles) and 6 motoneurons whose bias is also set by the brain model (see Figure 3). Figure 8-3: Distribution of gains among the various interneurons and muscles Gains are distributed into categories depending on whether they modulate the inputs or outputs of a given interneuron. Each category is subdivided further for monoarticular and biarticular muscles, as their connectivity with the rest of the muscles differs. 191 Brain model The brain is modeled as a task planner that evaluates performance according to criteria defined for each task and an adaptive controller that adjusts its control inputs based on performance (see Loeb, Brown et al. 1999). The control inputs (normalized for the range -1 to 1) set the bias of interneurons and motoneurons (which have sigmoidal input/output functions) and the gains of the local projections within the spinal cord. The learning scheme for the adaptive controller has been described in detail in Raphael et. al. 2010. Briefly, each control input is initialized at random within a relatively low range (-0.3 to 0.3) to avoid instability. The inputs are then tuned through a simple gradient descent process in which each gain is sequentially adjusted in the positive and negative direction and then left at the value that produces the best performance. One cycle through all the gains corresponds to one iteration. The model in this study was trained for three iterations and the size of the adjustments was 0.2, 0.2, and 0.1, respectively. Only three iterations were performed because they were sufficient for the model to converge on locally optimal solutions. Modeled task The response of a planar arm to a brief, oblique impulse (100N x 10ms) at the hand was simulated, equivalent to ~30Nm extension torque at each joint. The perturbation was applied at random between 0.5 and 1.5 seconds into the simulation to avoid anticipation and use of momentum in the opposite direction 192 rather than spinal reflexes to resist the perturbation. The gains of the SLR were initially set at random and adjustments were evaluated according to quadratic cost (squared deviation of the hand from the initial position integrated from 0.5 seconds before the perturbation to 2 seconds after). See Figure 4 for a schematic overview of the task. Figure 8-4: Overview of the modeled task An impulsive force (100N*10ms) is applied at the endpoint of a stationary arm at a random onset within a one second interval. Task performance is measured by integrating the displacement of the arm over two and half seconds (half a second before the perturbation to two seconds after). We also tested the response of the system when adding a modest level of cocontraction (20% activation to all muscles) to the SLR to simulate the experimental phenomenon where subjects cocontract more in the early phase of 193 learning and to examine the effects it may have on the adaptation process. The level of co-contraction was chosen to be as low as typically adopted by experimental subjects, which itself is insufficient to stabilize the arm on its own. To gain insight into its role once performance converges, we subsequently removed the cocontraction and ran an additional training session. Results Training the SLR without cocontraction led to generally poor performance (mean = 0.091, stdev = 0.15) that was often worse than applying cocontraction alone (SLR gains fixed at zero; Figure 5). Most of the converged solutions were mediocre with only two being acceptable (see exemplary solutions in Figure 5). The criteria for acceptable performance was derived from human subjects performing a similar task in Lacquaniti and Soechting 1986 (although they appeared to use a much slower and smaller perturbing impulse, which was not quantified). When cocontraction was added to the SLR, the system’s final performance was significantly better and less variable (mean= 0.006, stdev = 0.005). All of the converged solutions produced better performance than applying cocontraction alone, with over half of them being acceptable even by the strict performance criteria. 194 Figure 8-5: Learning curves with and without cocontraction Learning curves and exemplary joint angle trajectories for trials in which the spinal-like regulator was trained without (thin dotted line) and with co-contraction (thin solid line). The upper limit of experimental performance (derived from Lacquaniti & Soechting 1986; thick black solid line) and modeled performance achieved when applying co-contraction alone (thick gray solid line) is also plotted for reference. As shown in the learning curves in Figure 5, in both cases the initial cost did not correlate well with the cost of the converged solution. The trial with the best converged solution, for example, had one of the worst starting costs. Furthermore, most initial starting points for the trials with cocontraction had a higher cost than those corresponding to the system being subjected to cocontraction alone. Thus, 195 cocontraction by itself or added to a randomized SLR was not significantly better than the randomized SLR alone, but the addition of cocontraction to an SLR guaranteed that any randomized SLR would converge to a good performance with a modest amount of training. Removing the cocontraction signal from a converged, well-performing system produced an immediate deterioration in performance (mean = 0.242, stdev = 0.287) (Figure 6). Retraining the system using the previously converged solutions as starting points, however, achieved better performance (mean = 0.005, stdev = 0.005) than the SLR responses without cocontraction. Surprisingly, they were even slightly (but not significantly) better than the SLR responses with cocontraction (SLR + cocontraction: mean= 0.006, stdev = 0.005; retrained SLR following removal of cocontraction: mean = 0.005, stdev = 0.005). 196 Figure 8-6: Learning curves for systematic application and subsequent removal of cocontraction Learning curves for 10 trials in which the SLR was first trained with co-contraction followed by removing the co-contraction and further training (traces from left to right). The performance achieved with inactive muscles (first point) and cocontraction alone (second point) is also shown for reference. Discussion Role of co-contraction in learning novel tasks Co-contraction has been shown to be an effective strategy for stabilizing the arm in situations where external perturbations are applied (Lacquaniti and Soechting 197 1986; Franklin, Osu et al. 2003; Hasan 2005; Milner and Franklin 2005) or a high level of accuracy is demanded (Gribble, Mullin et al. 2003). Co-contraction has an obvious stabilizing effect because each muscle’s viscoelastic properties, termed preflexes (see Brown and Loeb 2000), intensify with activation. The effects of cocontraction on learning, however, are not as intuitive because understanding them requires knowledge of the control points (e.g. descending inputs and reflex gains) in the sensorimotor system, their influence on the performance criteria of a given task (defined largely by the neural connectivity), and the means by which they are adjusted (i.e. type of adaptive controller). Franklin et al. (2008) show that by adjusting the degree of feedforward antagonist muscle co-activation based on position feedback, it is possible to reproduce some characteristics of physiological learning of novel tasks. This purely feedforward scheme, however, does not take into account the actual control points of the nervous system and the nature of the solution space thereby afforded. Therefore, it provides limited insight regarding the actual control problem that the nervous system encounters and the opportunity to learn to replace energetically expensive cocontraction with well-chosen gains for proprioceptive feedback. Our results show that cocontraction reshapes the solution space, virtually eliminating the probability of entrapment in poor local minima. The local minima that are entered during learning with cocontraction are favorable starting points for learning to perform the task when cocontraction is abruptly removed. These results suggest that the tendency of subjects to resort initially to cocontraction when 198 learning a new task (Thoroughman and Shadmehr 1999; Franklin, Osu et al. 2003) may be an important step in the learning process that eventually results in mature strategies marked by little cocontraction and greatly decreased cortical activity. Role of the spinal cord The genetically specified and highly preserved connectivity of the mammalian spinal cord appears to provide a high dimensional control space that happens to facilitate rapid and successful learning of new motor tasks. This is true even when a simplified model of the spinal-like regulator is controlled by a highly oversimplified model of the brain and the algorithm by which it learns. The relatively simple and local optimization algorithm applied in this study was successful at finding many good-enough solutions because the state space defined by the SLR consists mostly of good local minima. The complex sequencing of muscle activations required to resist the perturbation in the absence of cocontraction were produced by the SLR circuitry itself acting on afferent and efference-copy signals generated by the perturbing torque and the subsequent responses of the SLR, according to well-chosen but unmodulated gains preset by the controlling brain. Limitations The model of the brain used in this study was purposely chosen to be highly simplistic to investigate the emergent properties of the SLR. As shown in this study and in Raphael et. al. 2010, the spinal cord appears to create a solution space that 199 facilitates learning “good-enough solutions” rapidly; these are properties that are obviously useful and are presumably exploited by the brain. The potentiality of spinal circuits may vary depending on the mechanics of the musculoskeletal system and the task. In the system described here, the added cocontraction signal (perhaps supplied by corticomotoneuronal cells in the biological system) was necessary at least initially to find good solutions with a high success rate. Interestingly, the SLR for a two degree-of-freedom but concentric wrist joint did NOT require any initial cocontraction to enable its controller to learn effective strategies to resist those perturbations and the solutions that it produced did not include cocontraction. It is possible that the requirement for initial cocontraction arises from the mechanical instabilities that arise in non-concentric multiarticular systems subject to Coriolis forces. If cocontraction is, indeed, an important feature of learning, then it would be useful for the brain itself to learn to apply and remove it according to the same learning algorithm used to adjust SLR gains. This can be accomplished by driving the learning process according to a cost function that includes both metabolic energy consumption and kinematic performance criteria. It may also be useful to employ a more biologically plausible learning algorithm in which the adaptive controller adjusts multiple SLR gains simultaneously instead of individually. These refinements of the model are currently underway and will be applied to the simple perturbation task described herein, as well as to the rich set of planar reaching tasks for which human performance data are available in the literature. 200 Implications for BMIs The tasks that we have chosen to teach our model systems are similar to those that have been chosen by most researchers developing BMIs for neural prosthetic applications. It seems plausible that retraining the brain to perform tasks that it used to perform with the intact spinal cord and musculoskeletal system will be easier and more successful if the prosthetic system incorporates properties and functions similar to those being replaced or bypassed. The circuitry modeled in the SLR appears to be complex but useful. It is easily emulated in software algorithms. A subject learning to use any BMI must be trained by asking the subject to imagine performing a particular task. The recorded neural activity can then be taken as the solution to the problem of controlling the SLR. Iterative algorithms could then be used offline to find a mapping between the various BMI outputs and the available SLR inputs that successfully performs the task. We hypothesize that such a control system will generalize more readily to the wider range of tasks that subjects must learn to perform, as opposed to BMIs mapped to outputs of the musculoskeletal system. 201 Chapter 9: A Realistic Model of Spinal Circuitry Facilitates Control of Center-out Reaching Movement George A. Tsianos, Jared Goodner, Gerald E. Loeb Preface The following manuscript will be submitted to a journal. It describes a refined model of the neurons incorporated in the SLR that accounts for the scalability of synaptic transmission, ultimately leading to improved learned rates and better performance of converged solutions. It also demonstrates that a model of spinal circuitry can generate all of the dynamics of muscle activity that is necessary for center-out reaching movement. The effects on learning when a realistic measure of metabolic energy consumption was included in the cost function were investigated for the first time; results show that the SLR model can be trained to produce movement that is both kinematically acceptable and energy efficient. Detailed kinematic and kinetic validation analysis illustrates the physiological behavior of the model. The model discovered many acceptable solutions to a given reaching task that were located in regions of the solution space marked by distinct local minima. Such property of the SLR increases the probability that acceptable solutions will be found rapidly and is likely to be exploited by the brain in the biological system. 202 Author contributions George Tsianos wrote the manuscript. Gerald Loeb edited the manuscript. George Tsianos designed and implemented the model of neuronal synaptic transmission. George Tsianos and Jared Goodner designed and implemented the modeling experiments. Gerald Loeb provided guidance and feedback on all aspects of this work. Introduction Even for seemingly simple tasks like locomotion, the underlying neural activity that controls the movement is quite complex and difficult to understand. The precise sequencing and modulation of muscle recruitment can be controlled entirely by the spinal cord through central pattern generators that make use of the abundant cutaneous and proprioceptive information that is continuously fed back to the spinal cord (Stein 1978). The spinal cord is clearly important for locomotor behavior, but its role in voluntary movements (such as made by the human arm) has long been debated (Hallett, Shahani et al. 1975; Scheidt and Rymer 2000; Ghez and Sainburg 1994; Loeb, Brown et al. 1999). It has often been argued that most of the dynamics of muscle recruitment are preprogrammed and are specified explicitly in neural signals descending from various areas in the brain. Such hypotheses are founded on past experiments showing that crisp sequencing of muscle activity occurred for ballistic movements in which there was insufficient time for feedback signals to influence the motor program via the brain. That fails to consider the local 203 feedback circuits in the spinal cord, which are now known to be much more broadly connected than merely providing servocontrol loops for individual muscles. Efference copy and proprioceptive signals can influence even the fastest movements because they could be arising from postural muscles that are activated substantially prior to the onset of any overt motion. For most physiological movements, feedback from a large number of muscles including those directly responsible for the desired movement is continuously integrated with descending commands through spinal interneuronal pathways (Pierrot-Deseilligny and Burke 2005). The highly divergent and convergent, nature of spinal circuitry tends to make the control problem look intractable. One alternative to simply ignoring it is to postulate that it results in a set of constraints or synergies by the brain that reduce the dimensionality of control (see Tresch and Anothy, 2009 for review). Muscle synergy theory assumes that the nervous system controls muscles in groups. It dates back to Bernstein (1947; english translation 1967), who speculated that this strategy would solve the problem of motor redundancy, in which there are more muscles than necessary to control the degrees of freedom of the skeleton. Synergistic groups of muscles are controlled simultaneously but with a different activation input as a function of time in response to a single command. A synergy is defined as one set of trajectories of muscle excitation that when combined with other synergies, typically through linear summation, reproduces a wide range of motor behaviors. Synergies can always be found in the EMG data from a restricted set of tasks, but they tend to break down when they are applied to new tasks, which 204 may involve new performance criteria such as complex trajectories, energy efficiency or end-point stiffness (Osu and Gomi 1999). Rather than constraining the system in an arbitrary way, it is possible that motor control strategies are learned by trial-and-error in which incremental adjustments are made to learn slightly different tasks. In this case, learned muscle activations for a certain class of tasks would have a similar character, tending to preserve the relative recruitment from the redundant set, and it would appear as if they were a result of combining synergies. Even if the nervous system could identify the appropriate balance of synergies, the command to the synergy is a purely a theoretical construct and is not at all clear how it relates to the anatomical components of the nervous system. In this study, we attempted to get insight into how the nervous system generates motor behavior by accounting explicitly for the anatomical components that comprise the actual plant being controlled: the musculoskeletal system and spinal circuitry. Making voluntary movements requires proper recruitment of muscles that exert torques at the joints. The brain controls these torques indirectly by sending commands to the spinal circuitry, which continuously integrates them with proprioceptive feedback and recurrent projections from motoneurons. Our model of the spinal circuitry is called a spinal-like regulator (SLR) by analogy to multi- input-multi-output engineered systems that appear to have similar functionality (He, Levine et al. 1991). The nature of this transformation and its implications for motor control have been investigated by building a realistic model of the lower motor 205 system (spinal circuitry plus musculoskeletal system) and determining how much of the dynamics of reaching movement it can generate entirely on its own. It has been shown that a model of the lower motor system of the wrist can be trained easily to perform a wide range of tasks physiologically by simply tuning the gains of interneuronal pathways and setting the biasing activity of interneurons (Raphael, Tsianos et al. 2010). This model, however, is relatively simple because the modeled musculature was not kinetically redundant and its mechanics were not influenced by interaction torques because its joints were concentric. These properties are present in musculoskeletal systems throughout the body and have been traditionally thought to pose major control challenges to the brain. The elbow- shoulder musculoskeletal system constrained to move in the horizontal plane was chosen for this study because it has all of these properties and it is a popular paradigm in the literature due to the convenience of its experimental setup. This system also has a larger set of muscles to be controlled which poses yet another challenge. In the biological system each muscle is accompanied by a set of spinal circuits with projections that interconnect circuit elements from other muscles. All of these projections are under descending control so each additional muscle contributes proportionally more tunable inputs. Despite the large number of control points in the spinal cord model (greater than 400 for our six-muscle model) and the oversimplified descending inputs (unmodulated step functions), it was surprisingly easy to train the system to perform center-out reaches to multiple directions along with the complex muscle dynamics required to achieve them. 206 Methods Phasic muscle excitation patterns that drive movement arise from interactions among many levels in the nervous system. A simulation environment was created that incorporated detailed models of the lower motor system (spinal circuitry plus musculoskeletal system) and an oversimplified model of the brain in order to force the lower motor system to generate all of the necessary dynamics. This system possesses a refined model of the neuron presented in Raphael et al. 2010 that incorporates scalability so that it is applicable to a larger range of musculoskeletal systems. Musculoskeletal system The musculoskeletal model has been described in detail in Tsianos et al. 2011. Briefly, it consists of an elbow and shoulder joint that link the trunk (grounded), upper arm and lower arm. Each joint is actuated by a pair of antagonist muscles that provide either flexion or extension torque in the horizontal plane. The model also has two biarticular muscles, one providing flexion and the other extension torques across both joints. Realistic muscle spindle models responding to muscle stretch and fusimotor control (Mileusnic, Brown et al. 2006) and Golgi tendon organs responding to muscle tension (Mileusnic and Loeb 2009) provided continuous input to the model of spinal circuitry. 207 Model of spinal circuitry The interneuronal pathways that comprise the model are detailed in Raphael et al. 2010, while their specific connectivity for the elbow-shoulder musculoskeletal system is described in Tsianos et al. 2011. Briefly, the model of spinal circuitry is based on a set of connectivity rules that have been derived from a combination of electrophysiological and anatomical studies. As shown in Figure 1, the neural network of the spinal cord has been divided into five classical pathways as defined by their homonymous and heteronymous connectivity, which is different among synergists and antagonists. The monosynaptic Ia pathway excites alpha motoneurons directly while the rest of the modeled pathways influence motoneuron activity through the following interneuronal types: propriospinal, reciprocal Ia- inhibitory, Renshaw inhibitory and Ib-inhibitory. Synergist and antagonist projections are such that a given peripheral stimulus of a particular muscle has the same effect on synergist muscles and the opposite effect on antagonist muscles. The circuitry was identified typically between pairs of muscles that are functional synergists or antagonists and has been adapted for the more common case where a pair of muscles can have both relationships depending on the nature of the task. We call this relationship partial synergist, which is thought to be mediated by spinal circuitry having both synergist and antagonist projections. The monoarticular muscles at each joint are modeled as strict antagonists while all other muscle pairings in the model system are modeled as partial synergists(Tsianos, Raphael et al. 2011). Each interneuron and pathway in the model is under descending 208 influence to model the effects of direct interneuronal drive and indirect modulation of transmission through presynaptic facilitation and inhibition(see Model of the neuron; Figure 2). Figure 9-1: Five classical interneuronal pathways that comprise the model from the perspective of a single muscle Projections from neural elements associated with self (HOM) as well as synergist (SYN) and antagonist (ANT) muscles are shown. Model of the neuron The motoneuronal output of spinal circuitry that excites muscles in response to input signals from the brain and periphery depends not only on the connectivity of the circuitry but also on the biophysical properties of the constituent neurons – the 209 fundamental computational elements of the nervous system. Neuronal axons influence the output spike train of their target neuron through a series of processes. Prior to reaching the synapse, activity within a given axon is modulated at the presynaptic terminal by spinal circuits that are themselves modulated by descending neural input. The resulting activity then causes current to flow into or out of the target cell through the process of synaptic transmission. The input current is accompanied by a change in membrane potential and can lead to output spiking activity depending on the properties of the cell (see Figure 2 for a schematic illustrating these processes and their computational implementation). Our approach to modeling neurons aims to capture only the major computational properties of the neuron while minimizing the number of arbitrary parameters and computational load. 210 Figure 9-2: Overview of the neuron model Schematic presentation of the major physiological processes contributing to interneuronal output and corresponding block diagram that illustrates their computational implementation. Note that the gain ‘S’ in synaptic transmission is positive for excitatory input and negative for inhibitory input. This model also applies to motoneurons but with interneuronal control (IC) equal to zero. Firing rate versus explicit spikes In general, the higher the firing rate of a given neuronal input, the larger the effect on the firing rate of the target neuron’s output. The higher the firing rate, the more neurotransmitter is released at the synapse over time, leading to more pulses of synaptic current entering the cell. If these currents are excitatory, for example, then positive ions will flow inside the cell and accumulate over time. Between 211 spiking events, however, the ion channels close and the net transmembrane current reverses direction to restore the resting membrane potential. At high enough rates, there will be a net inward current over one spiking period and the membrane potential will eventually exceed threshold and an action potential will be initiated. Following the action potential, the axonal membrane undergoes a refractory period where it is hyperpolarized to a large degree, making it unexcitable. The membrane potential gradually rebounds requiring progressively lower input currents to exceed its threshold and fire another action potential. Therefore, the larger the excitatory input current, the more depolarized the membrane potential, and the higher the firing rate of the neuron, but the relationship saturates as output rate approaches the inverse of the refractory period. This relatively simple transduction property is prevalent in neurons of the spinal cord. We can thus use a simple mathematical relationship that computes axonal output by summing the firing rates of all input axons rather than integrating post synaptic potentials generated by individual spikes. It is worth noting that many classes of neurons in the brain exhibit subthreshold oscillations of membrane potential and are therefore more sensitive to intermediate frequencies of stimulation whose bandwidth can be very narrow. In these situations it is important to model spike timing of the various inputs explicitly as opposed to a mean firing rate. Spike timing is also important when synaptic contacts are located close to each other. If a given synapse on a dendrite is active and its corresponding ion channels open, for example, subsequent activation of a more distal synapse would produce minimal changes to the neuron’s membrane 212 potential because the open ion channels nearby would create a shunt, thus, reducing the local input impedance. This effect is not captured in our model because of the lack of data on the location of individual synapses in the spinal cord. There is also a lack of evidence for systematic spatial patterns of projections for a given population of neurons, suggesting that even if these spike-timing effects existed at the neuronal level, they would be averaged out by the population. Presynaptic inhibition/facilitation The spike train within the incoming axon will release different amounts of neurotransmitter in the synaptic cleft depending on the level of presynaptic inhibition/facilitation it experiences. This modulatory effect is mediated by neuronal (axoaxonal) projections to the presynaptic terminal whose electrical excursions affect the influx of Ca 2+ , hence vesicle fusion and neurotransmitter release. These presynaptic gains are all assumed to be under direct or indirect cortical control and they constitute the majority of the cortical outputs that are adjusted as the controller learns to perform specific movements, but they are not adjusted dynamically when the movement is initialized or as it is performed (see below). The maximal firing rate of a given axon is finite and depends on the duration of the action potential plus refractory period. For simplicity, maximal firing is set to 1 in the model. The activity at the neuron’s presynaptic terminal is under descending control and is modeled by modulating the firing rate. Presynaptic control (PC) input 213 ranges from -1 to 1 to allow for either inhibitory or facilitating effects and is defined as a variable parameter in the model. The resulting signal represents the effective activation at the presynaptic terminal and lies within the range 0 to 1. This is accomplished by adding the afferent and modulatory signal and feeding their sum to a sigmoid function with a range between 0 and 1. Synaptic transmission Neurotransmitter molecules from the presynaptic terminal diffuse across the synaptic cleft and bind to ligand-gated ion channels on the postsynaptic membrane. Ion channels open upon binding and allow the flow of specific ions along their electrochemical gradient that generate synaptic current that alters the membrane potential. The extent to which the membrane potential changes depends on the input impedance of the cell, which further depends on the size of the cell and location of the synaptic contact. This simple, distributed model of cellular electrophysics seems to capture the most salient properties of those spinal neurons that have been studied, although it is clearly inadequate as a computational model of many neurons in the brain that have local circuit features such as dendritic spines (Polsky et al. 2004). The effect of an input current on the membrane potential of a given neuron is automatically scaled in the biological system according to the size of the cell, as first described for alpha motoneurons of the spinal cord (Henneman). Similarly, the influence of each input to the model cell is scaled by the number defined by “s” (see 214 Figure 2) that depends on cell size, defined in the model as the total number of inputs to the cell. Rather than “s” being an arbitrary parameter with some relation to the total number of inputs, it can be framed in terms of the maximal allowable hyperpolarization and overdrive of the neuron. For example, a given neuron receiving n exc excitatory inputs and n inh inhibitory inputs (each occupying a range from 0 to 1) with synaptic transmission gain of 1 would experience a membrane potential ranging from –n inh to n exc. It is sensible to scale the inputs such that they always occupy a reasonable range; otherwise, neurons with many inputs would tend to saturate easily. If the range is defined as –HYP to 1+OD, where HYP is the allowable hyperpolarization and OD is the allowable overdrive relative to the membrane potential that saturates the output, then this would require the following two scaling factors: š B�F = − ¡¢ � L£Q ; š …Œ? = M¥¦§ � ˆ¨I . The membrane potential that saturates the output is defined as 1 as shown in the sigmoid function presented in Action potential generation. Although separate synaptic transmission gains for inhibitory and excitatory inputs is unphysiological, it is sensible to assume that each neuron is designed to have the right balance between excitatory/inhibitory influences and input impedance such that its membrane potential lies within its operating range and away from the saturation limits. It is also likely that the actual number of neurons within each population modeled varies, which would warrant the use of different scaling factors to 215 compensate. Each of the interneurons in our SLR model actually represents a population of similarly connected but asynchronously active interneurons in the spinal cord. Action potential generation Depending on the intrinsic dynamics of the cell and resistive paths between individual synapses and the cell body, incoming synaptic currents are integrated into a cell membrane potential which is then converted to a spike train depending on the threshold and refractory properties of the axon potential initiation site near the axon hillock. Neurons within a population having a variety of membrane potential thresholds and firing sensitivities collectively have a sigmoidal response. Just like activity of the input axons, the output also ranges between 0 and 1. The sigmoidal relationship therefore accepts the summed effect of inputs representing membrane potential (see Synaptic transmission) and outputs a value from 0 to 1. The sigmoid function has the physiological attribute that a given change in input leads to different changes in output depending on the level of background activity, which defines the position of the system on the sigmoidal input-output function. The output firing rate of a given neuron in the model was computed using the equations below. Besides some number of excitatory and inhibitory projections to interneurons that arise either from proprioceptive afferents, other interneurons and alpha motoneurons, there is also an input called interneuron control (IC) that 216 represents the lumped influence of all descending signals on that interneuron. This value ranges between -1 and 1, as the lumped effect of descending signals can be either excitatory (the likely sign of direct corticospinal projections) or inhibitory (reflecting effects mediated by inhibitory interneurons under cortical control). ™ž˜©ž˜ = šœª «š …Œ? ∗ ¬ k—_œ¯©ž˜ B + š B�F ∗ ¬ œ¯ℎ_œ¯©ž˜ ± + ²y � L£Q ±!M � ˆ¨I B!M ³ where, šœª = M M¥… m´´¨mµ.¶ k—_œ¯©ž˜ B = šœªi B + ·y B ) œ¯ℎ_œ¯©ž˜ ± = šœªi ± + ·y ± Table 9-1: Symbols and definitions Symbols Definitions output Axonal output activity of a given cell (0-1) A Neural activity within afferent axons from proprioceptors, interneurons, or alpha motoneurons prior to presynaptic modulation (0-1) PC Descending drive that presynapticallyupregulates (>0) or downregulates (<0) the nominal activity of a given axon (-1-1) IC Descending drive that controlsinterneuronal activity directly (-1-1) exc_input Magnitude of total axonal activity after presynaptic modulation that excites the target neuron (0-1) inh_input Magnitude of total axonal activity after presynaptic modulation that inhibits the target neuron (0-1) s exc Scaling factor applied to exc_input to account for synaptic transmission (a.u.) s inh Scaling factor applied to inh_input to account for synaptic transmission (a.u.) n exc Total number of excitatory inputs to target neuron n inh Total number of inhibitory inputs to target neuron OD Maximum amount a target neuron can be overdriven (relative to input that leads to output saturation) HYP Maximum amount a target neuron can be hyperpolarize (relative to input that leads to output saturation) sig Smooth saturation function that maps total input to a neuron into its axonal output and modulatory effects of presynaptic control onto incoming axonal activity 217 Model of the brain The outputs of the brain model controlled fusimotor gain of muscle spindles, modulation of activity at each presynaptic terminal and biasing activity of interneurons. The modeled outputs did not project directly to alpha motoneurons, which is consistent with the proximal limb musculature modeled in this study. An oversimplified model of the brain was employed in order to force the lower motor system to generate all of the necessary dynamics. All commands to the spinal circuitry were unmodulated step functions. Every controllable element received a step input with its onset at the beginning of the simulation to set the background activity of the spinal cord prior to making a reach. The inputs projecting to the interneurons (interneuron control; IC) were subjected to an additional step function whose onset was coincident with the simulated ‘Go’ cue of the movement. The amplitudes of these step functions were tuned using a coordinate descent optimization method. The total number of parameters amounted to 438.They were initialized to random values and tuned individually in a randomized sequence until cost plateaued. Each amplitude was perturbed in the positive and negative direction by a predefined magnitude and was then set to the value that produced best performance. The model was trained to perform center-out reaches, initially in the outward left direction (see Figure 3). The endpoint had to follow a straight-line trajectory that reached the target in 0.5 seconds and maintained position for 1 second. The squared deviation from the desired trajectory, integrated over the entire duration of 218 the simulation, constituted the kinematic cost (in m 2 *s units). The energetic cost (in Joules) was the sum of metabolic energy consumed by all muscles in the set over the entire simulation. The energy consumed by each muscle was computed using the model presented in Tsianos et al. 2012. All descending inputs to the spinal circuitry model were initialized to random values within a range spanning -0.2 and 0.2. Adjustment of a single input marked one iteration and cycling through all inputs once corresponded to a learning cycle. The perturbation magnitude applied to each input was 0.2, 0.2, 0.1, and 0.1 for four cycles, respectively; performance typically plateaued by this point. The inputs were perturbed by large amounts in the beginning to avoid entrapment in poor regions of the solution space and were subsequently reduced to hone in on the local minimum. Initially, the system was trained using a purely kinematic cost function and if converged performance was acceptable, training was continued with energetics added to the cost function. Kinematic cost and energetic cost use different units; it was determined through systematic analysis that scaling the energetics term by a factor of 2x10 -4 caused them to have comparable effects on the learning process. The acceptable limits of kinematic and energetic cost were chosen such that a solution within these limits tended to produce both a nearly straight reaching trajectory that stabilized on the target and the muscle activity that had a phasic burst pattern marked by low amounts of cocontraction. 219 Results Exemplary solution to a reaching task The SLR model was trained successfully to reproduce the dynamics of a rapid point-to-point reach (Figure 3). This requires a tri-phasic burst pattern of muscles in order to generate agonist torques to accelerate the limb toward the target, antagonist torques to decelerate it and corrective torques to stabilize it on the target. Note that temporally modulated muscle activity must be produced by the SLR in response to unmodulated step functions applied only to the interneurons; thus the precise timing of the excitatory bursts is generated entirely by distributed proprioceptive feedback and its nonlinear integration by spinal circuits. Figure 3 shows the dynamical behavior of all feedback to the spinal circuitry model, namely muscle spindle Ia, GTO Ib and alpha motoneuron activity, which inhibits itself via the Renshaw pathway. 220 Figure 9-3: Exemplary reach to the outward-left target. 221 Top plot shows the path of the reach in Cartesian space with circles spaced 50ms apart in time. The thick black bars represent the initial configuration of the arm. Colored lines represent paths of the six muscles and match the colored traces below. The remaining plots show hand speed and alpha motoneuron, spindle Ia, and Golgi tendon organ Ib activity, all aligned in time with the GO signal at 1s (dashed vertical line). Activity related to extensor muscles is plotted in the upward direction, downward for flexors. Learning curves The coordinate descent algorithm tended to converge on acceptable kinematic performance after three to four cycles (typical learning curve in Figure 4 left). In this particular case, but not always (see Variability of performance), the energetic cost of the movement was also within acceptable limits. A more compact way to visualize the learning curve is introduced in Figure 4 (right) that represents performance as a point in 2-D space, showing clearly how kinematic and energetic behavior coevolve with time. Small open circles in the figure are spaced one learning cycle (438 iterations) apart, that is, the trajectory bounded by a given pair of small circles corresponds to the evolution of performance over one cycle of the algorithm. From this perspective, the goal is to guide the trajectory rapidly into a good-enough region in which kinematic and energetic cost are both acceptable. 222 Figure 9-4: Exemplary learning curve Exemplary learning curve showing the evolution of kinematic and energetic performance over four cycles of the optimization algorithm (438 iterations per cycle).The plots on the left show the evolution of kinematics and energetics separately. Filled circles represent the best performance achieved at the point while open triangles represent performance at that state when one input is perturbed in the positive direction (upward triangle) or in the negative direction (downward triangle). Dotted horizontal lines correspond to acceptable performance. The plot on the right shows energetic versus kinematic cost with time denoted by small open circles that are spaced one cycle apart. Good-enough performance is marked by the region where both energetic and kinematic cost are acceptable as denoted by the vertical and horizontal dotted lines, respectively. Variability of performance The system converged to good-enough performance from many different Monte Carlo initializations. Figure 5a shows learning curves for eight different 223 representative initializations in which only kinematic cost was considered. All of these learning trials converged eventually to good-enough performance; however, the energetic quality of the solutions was not consistent. About half of the converged solutions had a reasonable energetic cost while the rest were energetically wasteful, exhibiting high levels of cocontraction. 224 Figure 9-5: Multiple learning curves and the effects of energetics on learning A) The evolution of learned performance is shown using a purely kinematic cost function from 8 different random initializations. Good-enough performance corresponds to the region in performance space where both energetics and kinematics are acceptable as denoted by the dotted lines. B) The evolution of learned performance is shown using a combined cost function (kinematics plus energetics) from the converged solutions in A. Exemplary alpha motoneuronal traces are shown before and after training on either side of the learning curves. 225 Effects of energetics on learning To investigate whether the SLR can be trained to generate the same kinematic behavior but with more consistently physiological patterns of muscle activation, the converged solutions were tuned according to a combined cost function that included an energetics term in addition to kinematics. Energetics and kinematics correspond to fundamentally different aspects of behavior so their relative weighting in the cost function cannot be determined a priori. As mentioned in Methods, relative weighting was adjusted by trial-and-error until acceptable results were obtained. Further training of the system with a combined cost function reduced the energetic cost substantially while maintaining acceptable kinematic performance (Figure 5b). The evolution of muscle activation strategies is shown on the sides of Figure 5b for two exemplary situations: one in which the energetic cost is marginally acceptable initially (right) and another in which it is clearly excessive (left). The resulting muscle activation patterns are marked by less cocontraction and exhibit a phasic burst pattern more clearly. Learning to reach in other directions The intertial properties of a linkage having nonconcentric joints that link segments of nonzero mass are nonisotropic. For example, the effective inertia when reaching in a direction that requires a large excursion at the shoulder, as opposed to the elbow, is relatively large because rotating the shoulder requires moving a larger inertial load (mass of forearm plus upper arm whose center is located far away from 226 the axis of rotation). Furthermore, at every point in the movement, motion about each joint is subject to interaction torques, which depend on arm posture, joint velocities and accelerations(Hollerbach and Flash 1982). The demands placed on torques and underlying muscle activity, therefore, vary substantially with direction (Graham, Moore et al. 2003). To test if the model can cope with these varying demands, it was trained to perform center-out movements to eight evenly spaced directions in space. One of the converged solutions shown previously (trial 1) was used as a starting point to compare the effectiveness of learning from a random initialization as opposed to tuning a solution from a similar task learned previously, which seems more physiological. For every direction, the simple optimization algorithm successfully tuned the cortical inputs to achieve acceptable performance (Figure 6). The muscle activation patterns that emerged also appear physiological despite use of a purely kinematic cost function. Furthermore, it took a smaller number of iterations to converge (avg = 418, stdv = 251) than the case in which each reach direction was trained from random initializations (avg = 939 iterations, stdv = 584). 227 Figure 9-6: Center-out reaches to multiple directions Exemplary reaches shown to eight targets spaced evenly along the perimeter of a 10cm radius circle centered on the initial position of the endpoint. Overlaid circles on reach paths are spaced 50 ms in time to denote the dynamics of the movement. Alpha motoneuronal traces of all six muscles are shown beside each target where extensor activity is plotted in the upward direction and flexor activity is plotted downward. The key is shown in Figure 3. Kinematics and kinetics validation In order to compare muscle activation strategies emerging in our model with those from the experimental literature, it is important to verify that the joint kinematics of these movements are at least qualitatively similar to those observed 228 experimentally. As mentioned in the previous section, kinematics can have large effects on the torques necessary to produce a movement due to their contribution to interaction torques. They also modulate muscle force substantially, which would require proper compensation at the neural level to achieve the joint torques required for a given task. Experimental kinematics and underlying kinetics of center-out reaching movement were compared to those that emerged in the model (see Figure 7). Such detailed analysis has only been performed on monkeys (Graham, Moore et al. 2003) so differences were expected due to the different inertial properties and dimensions of their limbs. Nevertheless, the topology of the arm is similar to our idealized human model and key features of kinematic behavior for simian reaching agree closely with human psychophysics, so musculoskeletal kinetics should be at least qualitatively similar. Figure 7 and 8 show a detailed analysis of kinematics and kinetics of the experiment and model, respectively. Note that the coordinate frame of the model has been rotated to match the experiment. In Figure 7a. The dependence of peak elbow and shoulder joint velocities are plotted as a function of reach direction and results are similar for the experiment and model. It has a figure eight shape (note that only half of the reach directions were simulated) whose peaks align roughly with the direction of endpoint motion when only that joint is rotated. Peak velocity directions differ by about twenty degrees and shoulder peak velocity is substantially smaller. The discrepancy between the orientations of peak joint velocities is small 229 and is probably due to the mechanical constraints imposed on the monkey’s arm motion by the apparatus. Peak velocity of the shoulder may be smaller relative to the elbow because the inertial mass of the upper arm relative to the lower arm may be larger in the model. Figure 7d shows examples of elbow and shoulder contributions to endpoint velocity in extrapersonal space. Note that the model trajectories are slightly curved, but such curvature is often observed in experimental subjects as well. The instantaneous joint velocities over various locations in the workspace are also similar, as shown in Figure 7b (elbow) and Figure 7e (shoulder). The coevolution of shoulder and elbow kinematics are also similar throughout movements as demonstrated in Figure 7c (joint angles) and 7f (joint velocity). Generally, making these center-out movements is accompanied by joint rotation and velocity at the two joints in the opposite direction. Moreover, the relative joint velocities were fairly constant and were reproduced by the model, with the exception of the rightward and leftward reaches. 230 Figure 9-7: Comparison of experimental and modeled kinematics 231 Kinematic features of the center-out reaching movement performed by a nonhuman primate (Graham et al. 2003) and the human-like model, are plotted side-by-side for comparison. A) Polar plot depicting peak joint velocity versus reach direction. B,E) Contour plots showing instantaneous angular velocity at the shoulder and elbow, respectively. C,F) Reaching trajectories plotted in joint angle and joint velocity space, respectively. D) Endpoint velocity vectors plotted in Cartesian space along four reaches in perpendicular directions. The contributions of joint angular velocity to endpoint velocity is shown halfway along each reach. The dependence of peak torques on direction for center-out reaching movements is also captured by the model (Figure 8a). The peak torque in each joint is not aligned with peak velocity, as observed in the experiment. Overall, larger torques were required at the shoulder to overcome the larger inertial mass associated with rotating the shoulder. The shoulder torque relative to the elbow is overestimated by the model probably because the inertia of the upper arm relative to the lower arm in the model is higher than the monkeys' in the experiment. Although the effect of movement direction on peak torque is similar for the model and experiment, there is a large discrepancy in the instantaneous torques generated at the elbow (Figure 8c). In particular, the torque profile of most reaches in the experiment exhibited little modulation as compared to the model. The movements that monkeys were instructed to make could have been slow with respect to the inertial properties of their limb, which would allow the torques to be applied more gradually. Furthermore, the monkeys’ passive elbow joint moments near the targets were substantial (Graham, Moore et al. 2003) and probably made a large contribution to the antagonist torque necessary to decelerate the limb. The 232 authors mentioned that the apparatus could have had a significant amount of viscous damping, which also would also contribute to this discrepancy. When the reach trajectories were plotted in joint torque space (Figure 8c), they did not have a simple pattern as observed when the trajectories were plotted in joint kinematics space (Figure 8c,f). This illustrates that both in the experiment as well as in the model, joint torque does not possess a simple relationship with kinematics as in single joint systems; instead, they have a complex relationship that reflects the influence of interaction torques on controlling kinematic behavior. It is also evident from this figure that shoulder torques relative to the elbow were larger in the model as discussed previously. 233 Figure 9-8: Comparison of experimental and modeled kinetics 234 Kinetic features of the center-out reaching movement performed by a nonhuman primate (Graham et al. 2003) and the human-like model, are plotted side-by-side for comparison. A) Polar plot depicting peak joint torque versus reach direction. B,C) Contour plots showing instantaneous joint torque at the shoulder and elbow, respectively. D) Reaching trajectories plotted in joint angle and joint velocity space, respectively. Timing and magnitude of initial agonist activity Besides validating gross characteristics of the movement such as kinematics and kinetics, it is also important to validate the neural strategies that underlie behavior. For humans, neural strategies are usually assessed at the muscle level through surface EMG recordings due to the complexity and invasive nature of recording from other areas in the nervous system. In principle, these recordings reflect the level of excitation delivered to each muscle and can therefore be compared against the alpha motoneuron signals that emerge in the model. The validation, however, is not so straightforward due to the following reasons: 1) Surface EMG signals suffer from cross-talk, as they are a composite of electrical potentials arising from all muscles near the electrodes. This can result in a poor signal-to-noise ratio of the recorded activity for a particular muscle of interest. It also makes it difficult to discern the contributions of a pair of muscles to a given task if they are located close to each other, such as the uniarticular vs. biarticular heads of the triceps brachii. For this reason, surface EMG is also impractical for muscles located deep underneath the skin, hence their contributions to an observed movement cannot be ascertained. 235 2) Even if the excitation levels of all muscles involved in a task were perfectly accessible, there is considerable variability in muscle recruitment strategies across subjects (see Loeb 1993) and even trial-to-trial variability within subjects as in the case, for example, where subjects switch recruitment among redundant muscles to avoid fatigue (Singh et al. 2010). 3) The idealized musculoskeletal model presented here does not account for each muscle individually, but rather lumps them into functional groups, namely mono- and biarticular flexors and extensors. This can complicate validation as muscles that are represented by these groups may be controlled independently by the nervous system in some situations. The above reasons make it difficult or even meaningless to compare the fine details of muscle recruitment in the model with experiments; however, there are gross characteristics of recruitment that should be similar. Karst and Hasan (1991) observed that there are general rules for muscle recruitment during center-out movement that apply across a wide range of initial positions and reaching distances and are even highly consistent across subjects as well. Specifically, they found that the initial direction of torque (flexor or extensor) produced at each joint varied systematically with the angle between the long axis of the forearm and the straight path to the target (angle denoted as ψ). Interestingly, the direction of initial joint torque did not agree well with the direction of torque needed to accelerate the endpoint directly toward the target. The relative timing and magnitude between 236 agonist bursts at the shoulder and elbow also varied systematically with ψ (see Figure 9, copied from Karst and Hasan 1991). The reaching directions studied in this model were converted to angle ψ based on their location with respect to the forearm and the agonist muscle at each joint for a particular direction was defined as the one with the earliest onset latency; muscles having low activity (below 10% excitation) or without a clear phasic pattern were automatically not considered as agonists. Agonist onset was defined as the point at which activity reached 10% of maximum excitation and was identified by a custom program and verified by visual inspection. As in the experiment, the magnitude of the agonist burst was calculated as the integral of activity from onset to 100ms later. The results are plotted in the same format and beside Figure 9 to facilitate comparison. As seen in the figure, model results are qualitatively similar to the experiment both in terms of the dependence of agonist selection on ψ and the dependence of relative onset and magnitude on ψ. One obvious discrepancy, however, is that the model does not reproduce the relatively early onset of shoulder agonists relative to the elbow. This difference is probably inherent in the design of the model. The GO inputs are step functions that are applied simultaneously to all interneurons. Such synchrony is not realistic (see Scott, 1997) and may contribute to at least part of the discrepancy of latencies at the muscle level. The model also does not account for nerve conduction delays, which are longer for the musculature around the elbow due to its more distal location. Additionally, the early activation of shoulder muscles in the experiment may also be related to preparatory activity for stabilizing posture in anticipation of 237 the movement, which was not necessary in our model because the position of the shoulder was grounded. 238 Figure 9-9: Comparison of agonist initial activity between experiment and model 239 Relative timing and magnitude of initial agonist activity potted versus direction, ψ, defined as the angle between the long axis of the forearm and the straight-line path between the initial endpoint position and the target. Results from human subjects (Karst and Hasan 1991) and the model are plotted side-by-side for comparison. Plot symbols denote the direction of active torque generated at the shoulder and elbow, respectively. Timing and magnitude of initial activity of individual muscles Wadman and colleagues (1980) reported timing and magnitude characteristics of individual muscles during the initial phase of movement. They investigated activity of the pectoralis major (corresponding to shoulder flexor in the model), deltoid posterior (shoulder extensor), triceps lateral (elbow extensor), the long head of the biceps, brachioradialis, brachialis (the last three are elbow flexors and are lumped into the elbow flexor muscle in the model). Activity from biarticular muscles having large moment arms about the shoulder was not measured in this experiment. In particular, these muscles are the long head of the triceps and short head of the biceps. Although the long head of the biceps also crosses the shoulder joint, its moment arm in flexion when the arm is in the horizontal posture is substantially less than short head (Itoi et al., 2008). Coincidentally, for most of the reaches in this set, the biarticular muscles in the model were rarely activated to significant levels, with the exception of the flexor for the inward reach (see earlier Figure 6). Interestingly, the long head of the biceps, brachioradialis and brachialis muscles in the experiment all had highly similar patterns of activation, which facilitated comparison with the elbow flexor in the model. Figure 10 shows onset and magnitude of agonist bursts versus direction 240 reported in the study. Note that from the three elbow flexors in the experiment, only brachialis activity was plotted, because it was representative of the rest. Moreover, although a wide range of reaching distances were investigated in the experiment, both timing and magnitude characteristics were similar qualitatively. Modeling results are plotted in the coordinate frame of the experimental results in Figure 10. Modeling results and experimental data for movement onset are similar (Figure 10). For directions in which a given muscle is an agonist, its onset is earlier than when it functions as an antagonist. The shapes of the polar plots, therefore, relate to the range of reach directions in which muscles act as agonists or antagonists. The agreement of the experimental and model data illustrates that the model captures this aspect of center-out reaching quite well. 241 Figure 9-10: Comparison of onset of muscle activity between experiment and model 242 The onset of muscle activity versus reach direction is shown for individual muscles of human experimental subjects (Wadman et al. 1980) as well as corresponding muscles in the model. Note that the plots from the experiment include data from several reach distances, which did not have a significant effect on their qualitative shape. Activity onset was measured with respect to an arbitrary reference in order highlight the relative onset across all directions. The magnitude plots are also similar for reaching directions in which muscles function as agonists but relate poorly for directions in which they are antagonists (Figure 11). This discrepancy is likely due to the shorter movement duration of the experimental reaches. Subjects were instructed to make reaches as fast as possible, which requires large antagonist bursts (followed by correctives) to decelerate the arm and stabilize it on target quickly. The model task had a longer flight duration (500ms versus 350ms for the same reach), which is still fast enough to require an antagonist burst, but it can be applied more gradually and efficiently. The integral of the antagonist burst relative to agonist activity over a brief time period, therefore, is expected to be less for slower movements, which is consistent with the discrepancy between experimental and modeling results. 243 244 Figure 9-11: Comparison of magnitude of initial muscle activity between experiment and model The magnitude of initial muscle activity versus reach direction is shown for individual muscles of human experimental subjects (Wadman et al. 1980) as well as corresponding muscles in the model. Note that the plots from the experiment include data from several reach distances, which did not have a significant effect on their qualitative shape. Magnitude was measured as the integral of muscle activity over 40 ms following its onset. Discussion Temporal structure of descending commands Because the descending commands were constrained to be unmodulated step functions in this study, all of the dynamics of the excitation signals to the muscles were driven by ongoing proprioceptive feedback throughout the movement. The ability of such a simplified control scheme to reproduce the detailed dynamics of center-out reaching movement suggests that the brain does not necessarily need to specify these dynamics in its commands; instead it can recruit the appropriate spinal circuits to do it by exploiting the naturally occurring feedback from the periphery. These circuits would have the added potential advantage of automatically generating relatively short-latency responses to perturbations that might arise during the execution of such movements. When inferring the role of motor cortical commands that drive a particular behavior, it is important to consider the effects of continuous feedback to the interneurons. The musculoskeletal plant and spinal circuitry form a closed loop system in which state 245 feedback (position and velocity from proprioceptors, force from Golgi tendon organs, muscle excitation signals from Renshaw cells) drives muscle recruitment more directly than commands from motor cortex. In pure feedback control, only feedback gains need to be specified, which take advantage of the intrinsic dynamics of the plant to stabilize it in a desired state. The job of the controller then reduces to simply specifying the desired state. In the biological system, it is likely that the brain determines the set points of interneurons and gains of feedback pathways over some time period to control the dynamics of the task and any responses to potential disturbances that the brain would not counteract effectively due to long cortical loop delays. The results of this study indicated that this is indeed possible to accomplish despite the highly nonlinear nature of musculoskeletal mechanics and spinal circuitry. In fact, it was accomplished easily by simple trial-and-error adjustment of the descending inputs, making it more plausible that the brain interacts with the spinal cord in this manner. The ability and ease with which the lower motor system can be controlled to generate the observable motor output dynamics should be considered when interpreting cortical activity, as it implies a substantially different relationship to behavior than if the brain controlled muscles directly. The step functions used to excite interneurons in the model may be inadequate for more complex movements that require abrupt changes in muscle recruitment such as reaching out to an object and then interacting with it. Indeed, Yakovenko and colleagues (Yakovenko, Krouchev et al. 2011) showed that cats reaching out to a 246 lever with their forelimb, manipulating it, and then returning to initial position can be broken to down into sequences of simple movements that are controlled by sequential changes in both the muscle recruitment and motor cortical activity. Thus, such movements in our model would necessarily require multiphasic commands to the interneurons, but less modulated than the muscle excitation patterns required to achieve them. Solution space afforded by the lower motor system Many different sets of inputs to the model of spinal circuitry produced acceptable performance. Random initialization of the model resulted in converged input sets that were located in substantially different parts of the solution space, as evidenced by the highly variable energetic cost of the movements reflecting variable control strategies. Figure 12a shows that random initializations lead to substantially different converged input values. Figure 12b illustrates that the various solutions for the same reach direction are actually further apart than the solutions for different reach directions that were derived from a common seed. (This has important implications for attempts to produce intermediate movements by interpolation of solutions, as discussed in the companion paper.) The lower motor system appears to be crafting a solution space with a high density of good- enough local minima. In fact, this property holds even when relatively strict performance criteria are imposed such as minimizing both kinematic and energetic costs (Figure 5b). Such high density of good solutions improves the likelihood and 247 speed that they will be found by any learning algorithm. It appears to be an attribute of the genetically specified and phylogenetically conserved circuitry of the spinal cord that facilitates the brain’s task of learning to produce a wide range of useful movements. 248 Figure 9-12: Solution space analysis A) Distribution of learned ‘GO’ input signals for the same reach target (outward-left). In each case the amplitudes of the ‘GO’ step function were tuned from randomly chosen initial values. Note: there were 24 ‘GO’ signals applied to the model of spinal circuitry, one for each interneuron. B) Euclidean distance between all possible pairs of solutions to the same reach but different initializations and between all possible pairs of solutions to reaches in different directions but same initialization. 249 As discussed in the results, about half of the converged solutions exemplified acceptable energetic performance despite the purely kinematic cost function used to drive learning. A similar result was seen for 2DOF wrist movements generated by a SLR (Raphael et al., 2010). This suggests that the structure of the spinal circuitry tends to favor these types of solutions, probably as a consequence of the large number of reciprocal inhibitory circuits. These solutions may also differ with respect to their utility when new learning new tasks. Using a solution to a reach having low energetic cost, for example, was a good initialization for learning to make reaches in all other directions. The subsequently trained reaches were acceptable not only kinematically but also in terms of energy consumption despite the use of a purely kinematic cost function. It appears that if the system is initialized in a part of the solution space marked by low energetics and a modest kinematic cost, it tends to preserve the low energetic cost throughout learning. This does not apply in the case where the system inputs are initialized to random values because the initial kinematic cost tends to be extremely high, consequently requiring large changes in neural strategy, which is likely to drive the system out of the low-energy subspace. Effects of performance criteria on learning In this study, training the system with a purely kinematic cost initially and subsequently incorporating energetics ultimately led to acceptable kinematic performance with physiological patterns of muscle activation. Training with a 250 combined cost function from a random initialization generally led to worse kinematic performance (results not shown), which suggests that varying performance criteria throughout learning has important effects on converged performance. Such initial consideration of kinematic criteria followed by larger emphasis on energetics is consistent with experimental observations showing that subjects tend to adopt high levels of cocontraction during the early phases of learning a new task and gradually reduce it once acceptable kinematic performance is attained (see Thoroughman and Shadmehr 1999, for example). Which aspects of spinal circuitry contribute to its intrinsic properties? It appears that spinal circuitry is a useful intermediary between the brain and the musculoskeletal system that facilitates control of movement. A key question that naturally arises is, what aspects of its structure contribute mostly to its desirable properties? The first step would be to check if a particular subset of connections is responsible for the behavior or if it is merely the fact that the system is over-parameterized. This can be tested by building a set of randomly connected networks with the same number of projections, but different interneuronal types, and checking if the spinal circuitry model performs better, which would imply that there are some unique properties associated with its connectivity that probably evolved for this reason. It is also likely that many of the presynaptic gains that made up the bulk of the descending commands that had to be explored by coordinate descent are not 251 necessary or even useful for some or even all movements. We are currently exploring the effects of systematically fixing some of these gains on the rates and success of motor learning. Identifying which gains these might be is not obvious, however. As pointed out by Marder and colleagues (Grashow, Brookings et al. 2010), a highly variable parameter in an over-parameterized system is not necessarily unimportant for circuit function, but rather in each case its setting could be compensating for other changes in the network. The analysis of variability would therefore have to be complemented by assessing the sensitivity of performance to changes in the converged inputs. Converged inputs that are highly variable and exhibit low sensitivity might be indicative of gains that simply don’t matter to the particular class of tasks being considered. On the other hand, converged inputs with high variability and high sensitivity suggest that the corresponding neural elements are important for the performance of the system. Limitations of the learning algorithm Although the coordinate descent algorithm employed in this study was sufficient for learning reaching movements within a reasonable time, its convergence rate is fundamentally limited by the number of parameters being optimized because it tunes them one-at-a-time. Musculoskeletal models with a more realistic number of muscles would require a vast number of iterations due to the vast number of spinal circuits associated with them and their descending control. In the biological system, it is unlikely that the brain tunes the gains of spinal circuits individually. Even if the 252 activity of a single corticospinal neuron could be changed selectively, it would influence the gains of many spinal circuits due to the divergent nature of its collateral axons. It is more likely that the brain adjusts all inputs at once. Given the complexity of the lower motor system, it is unlikely that the brain computes appropriate changes in neural strategy by inverting the plant. To do so, it would require an inverse model of both the combined spinal circuitry and musculoskeletal apparatus and the convergence patterns of all of the brain’s outputs onto the hundreds of programmable elements in the SLR. Instead the brain might more plausibly apply random changes to its complete output state, perhaps selecting the magnitude or general direction of those changes based on prior experience learning similar tasks. Such optimization does not guarantee convergence to the global cost minimum, nor even a local minimum, but that is not the aim of the biological system anyway. The goal is to find a good-enough solution as quickly as possible, which relies heavily on a high density of acceptable solutions in the solution space that appears to be a property of the lower motor system (Goodner, Tsianos et al. 2011; Loeb 2012). 253 Chapter 10: Motor Programs to the Spinal-Like Regulator are Interpolable George A. Tsianos and Gerald E. Loeb Preface The following manuscript will be submitted as a companion paper to the one presented in Chapter 9. It demonstrates that, despite the highly nonlinear and complex nature of the SLR, solutions to untrained reaching movements can be interpolated from learned solutions. This applies to reaches having different durations, directions, distances, and even magnitudes of a dynamic force-field perturbing the hand, namely the curl-field. It also demonstrates that interpolation is most reliable when the learned solutions are from the same training sequence, which suggests that the motor strategies that the brain learns are influenced heavily by memory and not just task performance. Generalization of solutions to the SLR improves the rate at which new tasks can be learned and reduces the storage capacity necessary for memorizing motor repertoires. Such favorable emergent property of the SLR model facilitates the task of learning to control movement and is likely to be exploited by the brain. 254 Author contributions George Tsianos wrote the manuscript. Gerald Loeb edited the manuscript. George Tsianos designed and implemented the interpolation experiments. Gerald Loeb provided guidance and feedback on the interpolation experiments. Introduction To generate a desired motion, the brain must determine the appropriate set of neural commands or motor program. If the motion is highly practiced then it will recall it from memory. Exactly the type of memory recalled, however, is poorly understood. In theory, the brain could be recalling an internal model of the plant that it is controlling, i.e. spinal circuitry plus musculoskeletal mechanics, and using it to compute the appropriate motor program for the specific task at hand. Conversely, it could be recalling an explicit motor program, stored for the specific task. If the task that an organism is faced with differs substantially from previous experience, then it would have to be learned. For the internal model scenario, this would involve tweaking or augmenting the existing model to account for the new behavior without compromising its validity for motor tasks learned previously. But how would the brain go about verifying this? It seems wasteful to simulate all learned movements after adjusting the internal model. Alternatively, if validity of the new internal model were to be tested on only a subset of the learned behaviors, then how would the brain go about choosing this subset? The brain will inevitably overlook an important behavior because it does not know how the structure of the 255 internal model relates to the motor tasks; that is what it is trying to learn. If the overlooked behavior happened to rely on the aspect of the internal model that was changed, then the brain would have to relearn it in the future when faced with a similar task. Given the nonlinearity of the plant being internalized, it may be difficult to adapt it to a form that can handle the task and even if it is possible then it is likely that another class of tasks will be accounted for more poorly as a result (Atkeson 1989) so this wasteful process of unlearning and relearning is likely to persist. It is evident, therefore, that although having an internal model of the plant is highly efficient for recalling motor programs and storing them, learning it is computationally wasteful and perhaps implausible given the complexity of the plant. The alternative scenario where individual motor programs are recalled instead of computed ensures that previously learned programs remain valid but clearly makes inefficient use of the brain’s capacity for memory. Regardless of the type of motor memory employed, the brain must learn behaviors by trail-and-error because it has no way of knowing the structure or intrinsic dynamics of the plant that it is controlling; it simply knows whether a given motor program produces the sensory feedback desired. This method, although crude from an engineering perspective, can be highly effective if the solution space afforded by the plant has a high density of good solutions across the range of tasks to be learned. In fact, a model of spinal circuitry plus musculoskeletal system (lower motor system) has been shown to have such favorable solution space (Raphael, Tsianos et al. 2010; Tsianos et al. in preparation). 256 If internal models are indeed not realized in the biological system, then the solution space ought to have yet another property: interpolability. It should provide the ability to perform new tasks by mixing learned solutions in a simple manner. This would reduce the number of different motor programs that would have to be stored for memorizing complete motor repertoires as well as the time need to learn new tasks. The previously developed model of the lower motor system for a planar arm (Tsianos, Raphael et al. 2011; Tsianos et al. in preparation) was used in this study to investigate whether interpolability of solutions is plausible for the biological system. Due to the highly nonlinear nature of the system and high dimensional solutions space, we hypothesized that a solution would interpolate well only if the learned solutions used for interpolation were near each other in space. The tasks themselves, however, could be quite different and the degree to which they can be different was investigated in this study. Interpolability was tested and confirmed for a center-out reach motion across a wide range of task parameters consisting of reach direction, distance, flight duration and magnitude of a dynamic perturbation applied at the endpoint, namely the curl force field. 257 Methods Model of the lower motor system The model of the musculoskeletal system of an arm constrained to move in the horizontal plane has been described in Tsianos et al. 2011. The model of spinal circuitry for this system has been described in Tsianos et al (in preparation) and its physiological justification is explained in Raphael et al. 2010 and Tsianos et al. 2011. Model of descending commands As discussed in Tsianos et al. (in preparation), unmodulated step functions applied only to interneurons were sufficient for generating center-out reaching movement along with the necessary phasic muscle activity to achieve it. Out of the 438 inputs, 414 of them were triggered at the beginning of the simulation to set the preparatory state of the spinal cord that maintains posture prior to making the movement. These inputs corresponded to supraspinal control of presynaptic inhibition/facilitation that set the gains of reflex pathways and fusimotor input to the muscle spindles that set the sensitivity of their response to stretch. The other 24 inputs were coincident with the simulated ‘GO’ cue and were responsible for initializing the movement. 258 Learning algorithm Appropriate amplitudes of the step inputs for a desired task were learned via the coordinate descent method described in Tsianos et al. (in preparation). Input amplitudes were initialized either at random or to values learned previously for a similar task (see Training sequence). The amplitudes were then tuned individually in a randomized sequence until cost plateaued. Each amplitude was perturbed in the positive and negative direction by a predefined magnitude, the simulations were repeated, and the input parameter was then set to the value that produced best performance (i.e. lowest cost). Training sequence It has already been shown that several solutions to the same reaching task can be learned from different initializations chosen at random. By training the system initially with a kinematic cost function and then incorporating energetics, the system converged to solutions marked by acceptable movement and physiological patterns of muscle activation. Although all solutions fulfilled the same task requirements, they appeared to be located in mutually distant parts of the solution space, presumably reflecting distinct local minima. To increase the chances that solutions would interpolate, we trained the system to learn a pair of tasks (T1 and T2) in a specific sequence. T1 was learned from a randomly initialized set of inputs while T2 was learned with the solution of T1 as a starting point. Further to ensure interpolability, the learning algorithm’s search 259 space (i.e. step size used when adjusting parameters) was as small as possible while still enabling acceptable performance for T2. In preliminary research, a useful set of perturbation magnitudes applied to the inputs during optimization was determined to be 0.1, 0.1, 0.05, 0.05, respectively for each learning cycle. Task T1 was always the same while T2 varied depending on the reaching parameter that was tested for interpolability. T1 was a reach in the outward –left direction (135 degrees ccw with respect to the horizontal axis; see Figure 1) over a 10cm distance, with a flight duration of 500ms. Reach direction, distance, duration and curl-field magnitude parameters of T1 were varied to generate T2. An intermediate task Ti was defined according to parameters that were halfway (linearly) between T1 and T2. Interpolability was assessed by variously weighting all output states associated with T1 and T2 and comparing the cost of that mean state with respect to the mean parameters of Ti. See Table 1 for a summary of the reaching parameters tested for interpolation and the learned tasks whose solutions were used to generate them. It is important to remember that the computational properties of the interneurons and the mechanical properties of the musculoskeletal system are all highly nonlinear. There is no a priori likelihood that any interpolation, much less a linear one, will be a useful strategy other than the general notion that any continuous system tends to be linear over sufficiently small parts of its operating space. The usefulness of interpolation as a means to minimize the number of different motor programs that must be stored depends essentially on the distance between tasks for which interpolation is useful. 260 The viscous curl-field is a perturbation applied to the endpoint whose magnitude is proportional to hand velocity and direction is orthogonal to the direction of motion (Shadmehr and Mussa-Ivaldi 1994). The curl-field direction used in this study was chosen arbitrarily to be in the clockwise direction, i.e. it was 90 degrees with respect to hand motion in the clockwise direction. Reaching in the midst of a curl-field is a popular experimental paradigm for studying motor learning and it is widely hypothesized that the brain builds an internal model of the field and computes the dynamic compensatory commands that are necessary to counteract its effects (see for example Conditt, Gandolfo et al. 1997). Table 10-1: Interpolation test summary List of reach parameters that were tested for interpolability as well as their specific values that were learned (T1 and T2) and interpolated (Ti). There are multiple acceptable solutions (S1) for T1 (Tsianos et al. in prep) that can be used as starting points for learning T2, thus, multiple pairs of sequentially learned solutions to T1 and T2.Each distinct pair of solutions is referred to as a training sequence. It was shown that the solutions to T1 were located in different 261 regions in the solution space, having different local minima, which would imply that the corresponding solutions for T2 would also lie in distinct parts of the solution space. The simulation protocol described above was designed based on the hypothesis that the training sequence from which the learned solutions originate affects how well solutions to new tasks interpolate. To test this hypothesis we checked the interpolability of the reach direction parameter by using T1 and T2 solutions that were obtained from different training sequences. The interpolated solutions obtained were then compared with already tested scenario where T1 and T2 solutions are from the same training sequence. As an additional verification that the different solutions for T1 lay far from each other in the solution space (or more precisely, are near different local minima), their solutions were combined as described in Interpolation method and their effects on performance were investigated. The hypothesis to be tested is based on the notion that pairs of interpolated solutions T1 and T2 that are located relatively close to each other in the hyperspace are more likely to interpolate well, at least for tasks that are sufficiently close to each other. Interpolation method Each solution to a task is comprised of a vector of 438 inputs. In theory, there are many ways that the inputs of solutions to the two learned tasks can be combined to obtain the solution to an intermediate task. Surprisingly, a very simple interpolation applied to each input element was often sufficient for generating an 262 acceptable solution to the new, intermediate task. The interpolation function was assumed to be the same for every input element and was a smooth monotonic function that ranged from one learned input value to the other. Rather than assuming specific forms of the function and testing their effectiveness for the task to be interpolated, all input values were swept systematically within the range of their respective learned values and the optimal setting was determined. In fact, the result of this method can be indicative of the appropriate interpolation function. For example, if the desired task parameter was half-way between the learned parameters and the optimal input parameter was also half-way in between the learned values, then this would imply that a linear interpolation function would be appropriate. The interpolation method was implemented as follows: 1) Solutions (S1 and S2) were obtained for a pair of learned tasks (T1 and T2) that differed only in terms of one reach parameter (see Training sequence). 2) For each corresponding input element in S1 and S2, i.e. S1(input k) and S2(input k), a set of n interpolated values that spanned the range S1(input k) and S2(input k) evenly were computed. This generates a set of n interpolated solutions that are all candidates for the new task (Ti) to be performed. 3) Model performance was computed for all n solutions and the solution producing the best performance or lower cost was considered optimal within the interpolation scheme. 263 4) If the performance of the optimal solution was better than good-enough, then the new task Ti was deemed interpolable. The number of interpolated solutions to be tested, n, was chosen to be sufficiently high to eliminate the possibility that an interpolated solution that led to good-enough performance was not tested. It was determined empirically that a value of 30 was appropriate. Note that even if an interpolation function that is appropriate consistently across all situations is not found, the fact that at most only 30 trials would be necessary to achieve acceptable performance makes this an extremely efficient learning process compared to starting from scratch. Results All reach parameters that were tested were found to be interpolable and the interpolated solutions generally preserved those aspects of the two learned reaches that were identical. The value of the parameter of the interpolated solutions being learned was bounded by the values of the learned solutions whether it was reaching distance, direction, or duration. Learned solutions for reaches that are 45 degrees apart can be used to interpolate the solution for reaching toward an intermediate direction. Figure 1a shows the performance of the best interpolated solution for one particular training sequence. The performance of this solution was well within the acceptable limit and the appropriate activation patterns to the muscles emerged from the interpolation. The solution to the intermediate reach was also interpolable for two other training 264 sequences tested (Figure 2a and 3a), suggesting that this result is robust. As illustrated by the alpha motoneuron patterns in these figures (Figures 1-3, part b) and the pattern of inputs (part c) to the model of spinal circuitry, both the learned solutions and interpolated solutions differed substantially across training sequences. Every interpolated solution that was generated using the technique described in Interpolation method exhibited similar speed and distance and the direction was bounded by the two learned directions (Figures 1-3, part c). 265 Figure 10-1: Interpolation results for reach direction – training sequence 1 A) Kinematics and alpha motoneuron patterns of the solutions to learned tasks (T1 and T2) spaced 45 o apart as well as the best interpolated solution to the new task (Ti). Overlaid circles on endpoint paths are spaced 50ms apart in time. B) Reaching trajectories of all 30 interpolated solutions. C) Values of all input parameters to the SLR for solution to T1 (S1 – blue), solution to T2 (S2 – green) and solution to Ti (Si – red). Black vertical lines connect the learned values for each input to emphasize the pattern of input values that are changed to accomplish the three tasks; isolated red dots correspond to inputs that were identical for all three solutions. Note that location of each red dot with respect to green and blue dots is the same for every input and reflects the interpolated solution that produced the best results. 266 Figure 10-2: Interpolation results for reach direction – training sequence 2 A) Kinematics and alpha motoneuron patterns of the solutions to learned tasks (T1 and T2) spaced 45 o apart as well as the best interpolated solution to the new task (Ti). Overlaid circles on endpoint paths are spaced 50ms apart in time. B) Reaching trajectories of all 30 interpolated solutions. C) Values of all input parameters to the SLR for solution to T1 (S1 – blue), solution to T2 (S2 – green) and solution to Ti (Si – red). Black vertical lines connect the learned values for each input to emphasize the pattern of input values that are changed to accomplish the three tasks; isolated red dots correspond to inputs that were identical for all three solutions. Note that location of each red dot with respect to green and blue dots is the same for every input and reflects the interpolated solution that produced the best results. 267 Figure 10-3: Interpolation results for reach direction – training sequence 3 A) Kinematics and alpha motoneuron patterns of the solutions to learned tasks (T1 and T2) spaced 45 o apart as well as the best interpolated solution to the new task (Ti). Overlaid circles on endpoint paths are spaced 50ms apart in time. B) Reaching trajectories of all 30 interpolated solutions. C) Values of all input parameters to the SLR for solution to T1 (S1 – blue), solution to T2 (S2 – green) and solution to Ti (Si – red). Black vertical lines connect the learned values for each input to emphasize the pattern of input values that are changed to accomplish the three tasks; isolated red dots correspond to inputs that were identical for all three solutions. Note that location of each red dot with respect to green and blue dots is the same for every input and reflects the interpolated solution that produced the best results. 268 Solutions to reaching movements with a distance that was between two previously learned distances were also interpolable (Figure 4). As shown in Figure 4b, the direction of all reaching paths of the interpolated solutions was roughly the same and their reaching distances were bounded by the distances of the two learned solutions. Note that for the longer reaching distance of T2, the tri-phasic burst pattern of muscle activity is necessarily more pronounced with a shorter duration between agonist peaks in order to traverse a longer distance over the same period of time. By combining S1 and S2, the interpolated solution to reach Ti exhibits a tri- phasic burst pattern with intermediate magnitude and duration. Figure 10-4: Interpolation results for reaching distance. A) Endpoint kinematics and alpha motoneuron patterns of the solutions to learned tasks (T1 and T2) from an exemplary training sequence as well as the best interpolated solution to the new task (Ti). Overlaid circles on endpoint paths are spaced 50ms apart in time. B) Reaching trajectories of all 30 interpolated solutions. 269 Acceptable solutions to reaching tasks with new flight durations were generated easily by combining solutions to slower and faster reaches, respectively (Figure 5). All interpolated solutions have the same direction and distance (not shown) and are bounded by the flight durations of movements T1 and T2 (Figure 5b). The solution to T2 is marked by lower muscle activation and phasic muscle activity that is more spread out in time. Surprisingly, by mixing the right percentage of S1 and S2 to the highly nonlinear SLR, the appropriate muscle activations could be obtained for Ti that were marked by intermediate magnitude and dynamics. 270 Figure 10-5: Interpolation results for reaching duration A) Tangential speed profiles and alpha motoneuron patterns of the solutions to learned tasks (T1 and T2) from an exemplary training sequence as well as the best interpolated solution to the new task (Ti). B) Tangential speed profiles of all 30 interpolated solutions. Using a solution to an unperturbed reach (T1) as a starting point, the model was trained successfully to perform reaches in the midst of a strong curl-field applied to the endpoint. Interestingly, the model was able to compensate for the strong and highly dynamic perturbation applied to the hand by simply changing the amplitudes of the inputs to the model of spinal circuitry; it was not necessary to modulate the descending signals at all to achieve the dynamic compensation. Moreover, the 271 solutions to different magnitudes of the curl-field also interpolated well. It was observed in particular that a solution for a free reach and a solution to the same target but with a strong curl-field can be used to interpolate a solution for a curl- field having half the magnitude (Figure 6). As expected, when the learned solution S1 drove the model when the clockwise curl-field was turned ‘on,’ the trajectory deviated in a clockwise direction with respect to the ideal trajectory. The deviation increased with the magnitude of the curl field. When the learned solution to the reach in the presence of a strong field (S2) was tested in a weaker field, the trajectory deviated in the opposite direction, the counterclockwise direction, and when the field was turned off completely, the deviation increased even more. This result is consistent with the “after-effects” phenomenon observed experimentally, which indicates that the compensation to the curl-field was a result of changes in the dynamics of the muscle activity instead of a simple stiffening of the joints. If the model was simply stiffening the arm by cocontracting the muscles to a high degree, then removing the curl-field would have less of an effect on performance. The fact that there is dynamic compensation at the muscle level, however, does not necessarily imply that those dynamics are controlled directly by signals at the cortical level. This is illustrated by the model’s ability to perform the task by simply changing amplitudes of the step inputs of descending signals. In this case the dynamic compensation arises entirely from proprioceptive and alpha motoneuronal feedback and its processing via interneuron elements of the spinal circuitry model. Interpolating a reach in the midst of a curl-field having a modest magnitude resulted 272 in acceptable performance that also exhibited a sophisticated and metabolically efficient muscle activation strategy that relied more on neurally mediated “reflexes” rather than cocontraction and “preflexes” (Brown and Loeb 2000). When the interpolated solution drove the model in the absence of any curl-field (T1 condition), the resulting path of the endpoint deviated substantially in the counterclockwise direction; when the stronger curl-field of T2 was applied, the path deviated in the clockwise direction. Note that unlike the after-effects observed experimentally, the endpoint did not eventually stabilize on the target, within the allowed duration of the task. This discrepancy arose probably because the model’s inputs were constrained to be unmodulated step functions and therefore could not issue corrective action as experimental subjects appear to do early in the movement (Shadmehr and Mussa- Ivaldi 1994). 273 Figure 10-6: Interpolation results for curl-field magnitude A) Endpoint kinematics of the solutions to learned tasks (T1/S1 and T2/S2) from an exemplary training sequence as well as the best interpolated solution to the new task (Ti/Si) are shown along the diagonal. Off-diagonal elements show the performance of a particular solution on an untrained magnitude of the curl-field. For example, S1/Ti and S1/T2 subplots show the system’s response to a curl-field prior to adaptation to different magnitudes of the curl-field. S2/Ti and S2/T1 show after- effects upon reduction and complete removal of the curl-field, respectively. Si/T1 shows after-effects upon removal of a moderate curl-field and Si/T2 shows the system’s response upon doubling the curl-field prior to adaptation. B) Tangential speed profiles of all 30 interpolated solutions. To test the limits of the system’s interpolability, interpolation was attempted using learned solutions to reaches spaced 90 o apart. In this case, interpolation was not as successful. As demonstrated by the exemplary result shown in Figure 7, the performance of even the best interpolated solution was worse than the acceptable limit, marked by reaching trajectories having inappropriate timing and excursion. As noted in the figure, however, learning did generalize to a large extent. The 274 performance of the interpolated solution to the new task Ti is certainly better than the learned solutions to T1 and T2. This level of generalization would only be useful if the interpolated reach enables a substantial savings as a starting point for subsequent trial-and-error learning. Indeed, training the model using Si as a starting point was substantially faster at reaching good-enough performance (avg = 67.75 iterations, std = 76.65) than initializing the model to S1 (avg = 226, std = 82.4). Figure 10-7: Interpolation results using reaches spaced 90 o apart A) Endpoint kinematics of the solutions to learned tasks (T1 and T2) from an exemplary training sequence as well as the best interpolated solution to the new task (Ti). Overlaid circles on endpoint paths are spaced 50ms apart in time. B) Performance of interpolated solutions represented as mixtures of learned solutions S1 and S2. To ensure successful interpolation it was necessary to use learned solutions that were obtained from the same training sequence. When learned solutions for reaches spaced 45 o apart were obtained from different training sequences, the resulting endpoint kinematics of the interpolated solutions were not bounded by the kinematics of the learned reaches as reported above. They varied wildly both in 275 terms of direction and reaching distance (Figure 8a). In fact, similar observations were made even when solutions to the same task, from different training sequences, were interpolated (Figure 8b). The results are consistent with the highly nonlinear nature of the whole plant (SLR plus musculoskeletal system). They support the hypothesis that solutions to the same task that were trained from random initializations are likely to be located near local minima that are distinct and therefore, using them for interpolation is likely to generate interpolated solutions that lie far from either favorable region in the solution space. 276 Figure 10-8: Effects of location of learned solutions in the solution space A) Endpoint kinematics of the solutions to learned tasks (T1 and T2) as well as all interpolated solutions to the new task (Ti). The solution to T1 was obtained from training sequence 1 and is denoted in the figure as S1/seq1 while the solution to T2 was obtained from training sequence 2 and is denoted as S2/seq2. Overlaid circles on endpoint paths are spaced 50ms apart in time. B) Endpoint kinematics of the solutions to a learned task (T1) obtained from training sequence 1 and 2 (denoted as S1/seq1 and S1/seq2, respectively in the figure). The remaining trajectories were obtained by mixing different percentages of the two solutions as described in Interpolation method. Discussion The solutions to learned tasks generalized to novel situations that were intermediate in nature as long as the learned solutions were from the same training sequence and the learned tasks did not differ greatly from each other. Importantly the ranges of task parameters over which interpolability was observed, while not unlimited, were substantial. This is a necessary condition for it to be feasible to use 277 a learned and stored repertoire of programs to generate the virtually unlimited set of motor behaviors of which humans are capable. Local generalization has been observed experimentally in subjects that were first trained to counteract a novel force field in one portion of the workspace and subsequently tested for movements that differed in direction (Gandolfo, Mussa- Ivaldi et al. 1996; Mattar and Ostry 2007) and distance (Mattar and Ostry 2010). In fact, as shown by the thorough analysis in Mattar and Ostry (2007), generalization was very weak for directions of reach that differed by more than 45 o from a single trained direction and was nearly nonexistent when the two reaches differed by 90 o . Subjects did a poor job of extrapolating the effects of the curl-field in other directions even if more reaching directions were added to the training set but were still greater than 45 o apart from the test reach. When the test reach was between a pair of trained directions that were 90 o apart, generalization was good but not complete. This is remarkably consistent with the incomplete but substantial generalization observed in this modeling study when learned reaches that were spaced 90 o apart. The generalization was evident from the kinematics of the endpoint that were in the direction of the untrained target (although they were slow and undershot the target) and also from the fact that the interpolated solution could be easily and rapidly tuned to achieve acceptable performance. To our knowledge, generalization has not been tested when the test reach lies midway between two reaches spaced 45 o apart. Given the approximately two-fold improvement in generalization between trained reach directions that were 90 o vs. 278 135 o apart (Mattar and Ostry 2007), it is expected that subjects would achieve nearly complete generalization, which is what we observed in this modeling study. Modeling results showed that the level of generalization depended on whether the learned solutions used for interpolation were obtained from the same training sequence. If the interpolability of solutions is indeed a property that is exploited by the brain to speed up learning and save storage capacity, then this would influence the brain in learning motor programs that are similar to those stored from previous experience even when it can adopt other strategies that satisfy the current task better (Ganesh, Haruno et al. 2010; de Rugy, Loeb et al. 2012). It appears that such incremental learning of motor repertoires is useful for fulfilling a larger goal of the nervous system that is more than simply completing the task at hand in an optimal way, but also considers the rate at which new tasks are learned and the memory capacity that they will require. 279 Chapter 11: Conclusions and Future Directions George A. Tsianos Conclusions The results of this research provide further evidence that the spinal cord together with the musculoskeletal system have emergent properties that facilitate the brain’s task of controlling movement. For both the wrist and planar arm musculoskeletal system, unmodulated step commands applied only to interneurons were sufficient to initiate the complex muscle dynamics necessary for a wide range of movements. Muscle recruitment strategies for a given movement could be modified depending on the energetic objectives of the task. There were many solutions or sets of inputs to the SLR that could accomplish a particular task, which should increase the likelihood and speed that any trial-and-error learning algorithm would attain good-enough performance. Furthermore, it was shown that solutions to new tasks can be interpolated from solutions to tasks learned previously. Such generalization of solutions improves the rate of learning in many situations and also reduces the storage capacity required for the brain to memorize motor repertoires. All of these emergent properties of the SLR are highly desirable and are therefore likely to be exploited by the brain when it learns to control motor tasks. They 280 should, therefore, be considered when assigning functionality to motor areas of the brain. Future Directions Intelligent coordinate descent The coordinate descent algorithm used in all of the studies reported here tuned each input sequentially by perturbing it to the positive and negative direction and setting the input to the value that produced the best performance. The magnitude of the perturbation applied was set prior to the onset of learning and was fixed throughout. Development of a more sophisticated learning algorithm in which the magnitude of the perturbation may be different for each input and depends on the sensitivity of performance to that input is underway (Sunwoo, Tsianos et al. 2012). For example, if the performance of a given input degrades substantially when it is perturbed in either direction, then this implies that the input is knocked further away from the local minimum and reducing the perturbation magnitude may bring it closer, thereby improving performance. Inputs having little influence on performance may be skipped during optimization, thus reducing the number of simulations needed for convergence and improving the rate of learning. It may also be beneficial to set the sequence in which inputs are tuned according to the sensitivity observed for all inputs in the previous cycle. 281 A physiologically plausible learning model Even unmodulated commands from the model brain tended to produce complex dynamic responses to the motor tasks studied. By setting an appropriate cost function and iteratively adjusting the individual inputs to the SLR, the coordinate descent algorithm could get the limb to generate biologically realistic behavior. But the individual corticospinal neurons have highly divergent terminations that affect many SLR inputs simultaneously, making it unrealistic to learn by adjusting those gains one at a time. Moreover, individual adjustment of inputs would require a vast number of iterations for more realistic neuromusculoskeletal systems having a larger number of controllable parameters. BioSearch is an optimization algorithm similar to simulated annealing that makes adjustments to all spinal circuitry parameters simultaneously and is driven only by information that we know is available to the brain: performance relative to the goal and the rate of its improvement (Goodner et al. 2012). When performance is far from the target level, the adjustments are drawn randomly from a probability distribution that has a large range of possible gain changes. The adjustments are also large if performance is improving very gradually. These strategies are designed to avoid entrapment in undesirable local minima. BioSearch uses this information to modify neural strategies on a trial-and-error basis, in which spinal circuitry parameters are modified and the effects on performance are evaluated. Each of the 438 parameters within the SLR model (each representing a gain or bias affiliated with a descending axon synapse) is initialized to some value, and the 282 parameter vector as a whole is considered a state. After running one simulation in the current state, wherein the musculoskeletal/SLR model is applied to the task described below, a cost value is extracted from a function that evaluates its kinematic and energetic performance. This cost value (C i+1) is utilized to calculate the approximate slope of the nearby landscape (C Δ = C i+1 - C i), as well as the distance from a user-specified goal cost (C* = C i+1 - C goal ). C Δ and C* are then utilized by a sigmoidal surface function to calculate the spread of a uniform distribution with which to build the vector that will generate the next neighbor state (Figure ). The neighbor state is generated by adding a vector of random values (generated by the uniform distribution) to the current state. If this neighbor performs better than the current state, the current state is replaced. The probabilistic nature of this step reflects the nature of the relationship between the brain and SLR, in that the brain is not capable of tweaking each gain individually, but utilizes sigma to send signals described by this adjustable distribution. This process of neighbor generation and evaluation is repeated as the algorithm navigates towards states with better costs. 283 Figure 11-1: BioSearch overview Exemplary learning curve and range of input changes applied (sigma) to current input vector depending on current performance relative to the goal (C*) and improvement of performance (C Δ ). BioSearch is capable of converging on acceptable performance for the stabilization task rapidly as compared to coordinate descent. BioSearch's success can be attributed to: 1) the manner by which it navigates the landscape and 2) the advantageous landscape generated by the SLR's neural network. BioSearch's hyperspace search capabilities stem from a strong sampling strategy. By perturbing every gain value at the same time, it is able to perform a wider search than the dynamic programming algorithm utilized previously. Random sampling is made more effective by the SLR's emergent properties, in that many "good-enough" solutions exist, and they exist all over the search space. By adapting perturbation size based on the surrounding search space (distance to goal performance, 284 difference from the last gain vector's performance), BioSearch is better able to fine tune task performance without invoking knowledge or mechanisms that are physiologically implausible. Extending the model to a 3D shoulder-elbow system Musculoskeletal models of non-human primates are useful to understand motor control areas of the brain and the role that the spinal cord plays in specific motor control tasks. In order to simplify the problem, most experiments and the modeling studies presented here have been limited to arm movements in the horizontal plane that can be modeled by a small number of muscles operating one degree-of-freedom (DOF) shoulder and elbow joints with no effects of gravity. Increasingly, researchers are studying more natural, three-dimensional (3D) reaches that require a three DOF shoulder joint, leading to the need of more complex musculoskeletal models (Chan and Moran 2006). Control of the additional skeletal DOFs requires additional muscles, so it will be important to extend the SLR and modeling results presented here to these larger systems. Although the force of gravity is constant, its effects on joint torques and necessary muscle forces to counteract them during a movement are dynamic. It would interesting to see if the simplified control scheme employed in this research is sufficient to cope with this complexity. 3D models designed with realistic but complicated anatomy are computationally inefficient and may be too complex to provide insights into the control strategies, hence, a simple and accurate model is needed. Arguelles and Loeb (2012) started 285 with a planar musculoskeletal primate arm that was modeled based on experimental data from the literature and existing models available for the macaca mulatta for muscles that cross the shoulder and elbow joints. The model was reduced to a minimal set of 6 muscles, 4 monoarticular and 2 biarticular. This 2D model was then extended for reaching tasks in three dimensions. This new model aims to be the simplest that can account for actual task performance, with 8 muscles (including 4 monoarticular and 2 biarticular muscles) for 4 DOFs: 3 DOF shoulder joint (including internal/external rotation, abduction/adduction and flexion/extension) and 1 DOF elbow joint. The pulling directions of some of these muscles must exert torques on more than one axis of motion and must include moment arms that depend complexly on posture. T he model was validated against the maximal torques vs. joint angles obtained from existing models of the complete musculature. Driving the SLR with cortical activity Neural activity from various areas of the cortex has often been related to various observable kinematic parameters in extrapersonal space. Because cortical activity correlates well with these parameters under some conditions, it has been thought that it encodes motor output and has been used to predict motor intent for various applications such as controlling prostheses. The relationship derived between the activity and motor output, i.e. the decode, however, applies only to highly constrained tasks, implying that neural activity represents something more complex 286 than any aspect of the observed behavior. Neural activity in the cortex reflects a combination of sensory feedback from the limbs as well as the properties of the neural structures is projects to. In reality, prior to influencing motor output, descending activity undergoes major transformations such as sensorimotor integration in the spinal cord that processes it into muscle excitation patterns and musculoskeletal mechanics that define the effects of the excitation onto the observable movement. Assuming that these transformations are represented accurately by the model presented here, then in theory, it should be easier to relate the cortical activity to the inputs of this system since they normally interact with it. One major challenge of such task, however, is that the precise mapping of the recorded signals to the model system is unknown and would have to be learned. Unless the controllable inputs to the spinal circuitry can be reduced to a tractable number, such a task may be computationally forbidding because the number of tunable parameters would be the product of the number of recorded signals and the number of control points in the spinal cord model. Although there is wide evidence for presynaptic modulation in the spinal cord, it is likely that it is not universally under descending control. Candidate classes can be inferred from modeling studies by analyzing the variability of converged inputs and the sensitivity of performance to changes in their values. The case in which variability of a converged input is low and sensitivity is high would suggest that the presynaptic input can be fixed to the mean converged value. Such change to the model would also reduce the control dimensionality in a useful way as it would 287 avoid wasting time exploring unfavorable areas of the solution space, where input value deviates from the appropriate setting. A converged input with high variability and low sensitivity can also be fixed, but the exact value would not matter. This would also decrease the dimensionality of the system, but would not necessarily be useful for control as its low sensitivity would not affect any decisions during learning. Care must be taken not to remove controllability from SLR gains that can be fixed or ignored for some but not all classes of tasks. The work presented here considered only a simple reaching task. Fundamentally different tasks such as generation of isometric force likely depend on different subsets of the SLR architecture (Raphael et al. 2010). Regulators for different classes of systems If the brain does not possess an internal inverse model of the plant to compute the appropriate motor program for the desired movement, then it would have to learn it by trial-and-error. The performance of any such algorithm depends on the structure of the solution space, that is, the system’s performance with respect to all possible combinations of input parameters. If the space has a high volume fraction of good-enough solutions that are widely distributed, it should be relatively easy to discover them. The structure of the solution space afforded by a plant can be altered by adding intermediate elements that process control and feedback signals prior to either of them influencing the plant. It appears that the genetically specified circuitry of the spinal cord is the bridge between the musculoskeletal plant and the 288 brain that facilitates control. Appropriate regulators for other plants such as robotic manipulators are probably substantially different due their different types of actuators and sensors. It would be interesting to test whether regulators for these systems can be evolved to facilitate their control just as the spinal circuitry appears to do this for the brain. It would particularly exciting to see if the evolution of the spinal circuitry can be recapitulated via programmatically evolving a regulator that crafts a favorable solution space for a model of the musculoskeletal plant. 289 References Allen, D. G., G. D. Lamb, et al. (2008). "Skeletal Muscle Fatigue: Cellular Mechanisms." Physiological Reviews 88(1): 287-332. Allen, G. M., S. C. Gandevia, et al. (1995). "Reliability of measurements of muscle strength and voluntary activation using twitch interpolation." Muscle & Nerve 18(6): 593-600. Alstermark, B., T. Isa, et al. (2007). "The C3–C4 propriospinal system in the cat and monkey: a spinal pre-motoneuronal centre for voluntary motor control." Acta Physiologica 189(2): 123-140. Alstermark, B., A. Lundberg, et al. (1984). "Integration in descending motor pathways controlling the forelimb in the cat." Experimental Brain Research 56(2): 293-307. Alstermark, B. and S. Sasaki (1985). "Integration in descending motor pathways controlling the forelimb in the cat. 13. Corticospinal effects in shoulder, elbow, wrist, and digit motoneurones." Experimental Brain Research 59(2): 353-364. Andersen, P., R. P. Adams, et al. (1985). "Dynamic knee extension as model for study of isolated exercising muscle in humans." Journal of Applied Physiology 59(5): 1647-1653. Arguelles, E. and G. E. Loeb (2012). "Musculoskeletal modeling for three dimensional reaching tasks." Grodins Symposium. Atkeson, C. G. (1989). "Learning arm kinematics and dynamics." Annual Review of Neuroscience 12: 157-183. Barclay, C., R. Woledge, et al. (2007). "Energy turnover for Ca2+ cycling in skeletal muscle." Journal of Muscle Research and Cell Motility 28(4): 259-274. Barclay, C. J. (1996). "Mechanical efficiency and fatigue of fast and slow muscles of the mouse." Journal of Physiology 497 ( Pt 3): 781-794. 290 Barclay, C. J., P. D. Arnold, et al. (1995). "Fatigue and heat production in repeated contractions of mouse skeletal muscle." Journal of Physiology 488 ( Pt 3): 741-752. Barclay, C. J., J. K. Constable, et al. (1993). "Energetics of fast- and slow-twitch muscles of the mouse." Journal of Physiology 472: 61-80. Barclay, C. J., G. A. Lichtwark, et al. (2008). "The energetic cost of activation in mouse fast-twitch muscle is the same whether measured using reduced filament overlap or N-benzyl-p-toluenesulphonamide." Acta Physiol 193: 381-391. Bashor, D. P. (1998). "A large-scale model of some spinal reflex circuits." Biol.Cybern. 78(2): 147-157. Berardelli, A., M. Hallett, et al. (1996). "Single-joint rapid arm movements in normal subjects and in patients with motor disorders." Brain 119: 661-674. Bernstein, N. A. (1967). Human Motor Actions: Bernstein Reassessed (Translation, edited by Whiting, H.T.A.), Elsevier. Bhargava, L. J., M. G. Pandy, et al. (2004). "A phenomenological model for estimating metabolic energy consumption in muscle contraction." Journal of Biomechanics 37(1): 81-88. Bolstad, G. and A. Ersland (1978). "Energy metabolism in different human skeletal muscles during voluntary isometric contractions." European Journal of Applied Physiology and Occupational Physiology 38(3): 171-179. Brown, A. G. (1981). Organization in the Spinal Cord, Springer-Verlag. Brown, I. E., E. J. Cheng, et al. (1999). "Measured and modeled proferties of mammalian skeletal muscle. II. The effects of stimulus frequency on force- length and force-velocity relationships." Journal of Muscle Research and Cell Mortility 20(627): 643. Brown, I. E., T. L. Liinamaa, et al. (1996). "Relationships between range of motion, L0, and passive force in five strap-like muscles of the feline hind limb." J.Morphol. 230(1): 69-77. Brown, I. E. and G. E. Loeb (1999). "Measured and modeled properties of mammalian skeletal muscle. I. The effects of post-activation potentiation on the time course and velocity dependencies of force production." J.Muscle Res.Cell Motil. 20(5-6): 443-456. 291 Brown, I. E. and G. E. Loeb (2000). "Measured and modeled properties of mammalian skeletal muscle: IV. dynamics of activation and deactivation." J.Muscle Res.Cell Motil. 21(1): 33-47. Brown, I. E. and G. E. Loeb (2000). A reductionist approach to creating and using neuromusculoskeletal models. Neuro-Control of Posture and Movement. J. Winters and P. Crago. New York, Springer-Verlag: 148-163. Brown, I. E., T. Satoda, et al. (1998). "Feline caudofemoralis muscle: muscle fiber properties, architecture and motor innervation." Exp.Brain Res. 121: 76-91. Brown, I. E., S. H. Scott, et al. (1996). "Mechanics of feline soleus: II. Design and validation of a mathematical model." J.Muscle Res.Cell Motil. 17(2): 221-233. Bullock, D. and S. Grossberg (1992). "Emergence of tri-phasic muscle activation from the nonlinear interactions of central and spinal neural network circuits." Human Movement Science 11: 157-167. Bullock, D., S. Grossberg, et al. (1993). "A Self-Organizing Neural Model of Motor Equivalent Reaching and Tool Use by a Multijoint Arm." Journal of Cognitive Neuroscience 5(4): 408-435. Burke, D., J. M. Gracies, et al. (1992). "Convergence of descending and various peripheral inputs onto common propriospinal-like neurones in man." The Journal of Physiology 449(1): 655-671. Butler, T. M., N. A. Curtin, et al. (1972). "Comparison of ATP usage in muscle during isometric tetanus and activated isovelocity stretch." Fed Proc 31: 337. Cavallari, P., R. Katz, et al. (1992). "Pattern of projections of group I afferents from elbow muscles to motoneurones supplying wrist muscles in man." Experimental Brain Research 91: 311-319. Chan, S. S. and D. W. Moran (2006). "Computational model of a primate arm: from hand position to joint angles, joint torques and muscle forces." J.Neural Eng. 3: 327-337. Cheng, E., I. E. Brown, et al. (2000). "Virtual Muscle: A computational approach to understanding the effects of muscle properties on motor control." J.Neurosce.Methods 101: 117-130. Churchland, M. M., A. Afshar, et al. (2006). "A central source of movement variability." Neuron 52(6): 1085-1096. 292 Churchland, M. M. and K. V. Shenoy (2007). "Temporal complexity and heterogeneity of single-neuron activity in premotor and motor cortex." Journal of Neurophysiology 97: 4235-4257. Cisek, P., S. Grossberg, et al. (1998). "A cortico-spinal model of reaching and proprioception under multiple task constraints." J.Cogn Neurosci. 10(4): 425-444. Conditt, M. A., F. Gandolfo, et al. (1997). "The motor system does not learn the dynamics of the arm by rote memorization of past experience." Journal of Neurophysiology 78(1): 554-560. Constable, J. K., C. J. Barclay, et al. (1997). "Energetics of lengthening in mouse and toad skeletal muscles." Journal of Physiology 505 ( Pt 1): 205-215. Crago, P. E., J. C. Houk, et al. (1982). "Sampling of total muscle force by tendon organs." J.Neurophysiol. 47(6): 1069-1083. Criscimagna-Hemminger, S. E., O. Donchin, et al. (2003). "Learned Dynamics of Reaching Movements Generalize From Dominant to Nondominant Arm." Journal of Neurophysiology 89(1): 168-176. Crone, C., H. Hultborn, et al. (1987). "Reciprocal Ia inhibition between ankle flexors and extensors in man." The Journal of Physiology 389(1): 163-185. Crow, M. T. and M. J. Kushmerick (1982). "Chemical energetics of slow- and fast- twitch muscles of the mouse." The Journal of General Physiology 79(1): 147- 166. Crowninshield, R. D. and R. A. Brand (1981). "A physiologically based criterion of muscle force prediction in locomotion." Journal of Biomechanics 14: 793-801. Curtin, N. A. and R. E. Davies (1973). "Chemical and mechanical changes during stretching of activated frog skeletal muscle." Cold Spring Harbor Symposia on Quantitative Biology 37: 619-626. Davoodi, R., I. E. Brown, et al. (2003). "Advanced modeling environment for developing and testing FES control systems." Medical Engineering & Physics 25(1): 3-9. de Rugy, A., G. E. Loeb, et al. (2012). "Muscle coordination is habitual rather than optimal." J Neurosci 32(21): 7384-7391. 293 Doyon, J. and H. Benali (2005). "Reorganization and plasticity in the adult brain during learning of motor skills." Current Opinion in Neurobiology 15: 161- 167. Eccles, J. C., R. M. Eccles, et al. (1961). "Electrophysiological investigations on Renshaw cells." Journal of Physiology 159(3): 461-478. Eccles, J. C., R. M. Eccles, et al. (1957). "Synaptic actions on motoneurons caused by impulses in Golgi tendon organ afferents." Journal of Physiology 138: 227- 253. Eccles, J. C., P. Fatt, et al. (1954). "Cholinergic and inhibitory synapses in a pathway from motor-axon collaterals to motoneurones." J Physiol 126: 524-562. Eliasmith, C. and A. H. Anderson (2003). Neural Engineering: computation, representation, and dynamics in neurobiological systems. Cambridge, MA., MIT Press. Enoka, R. M. and J. Duchateau (2008). "Muscle fatigue: what, why and how it influences muscle function." Journal of Physiology 586(1): 11-23. Evarts, E. V. and J. Tanji (1976). "Reflex and intended responses in motor cortex pyramidal tract neurons of monkey." Journal of Neurophysiology 39(5): 1069-1080. Feldman, A. G. (1966). "Functional tuning of the nervous system with control of movement or maintenance of a steady posture. II. Controllable parameters of the muscles." Biophysical Journal 11: 565-578. Fetz, E. E. (1992). "Are movement parameters recognizably coded in the activity of single neurons?" Behavioral and Brain Sciences 15: 679-690. Fetz, E. E., P. D. Cheney, et al. (1989). "Control of forelimb muscle activity by populations of corticomotoneuronal and rubromotoneuronal cells." Progress in Brain Research 80: 437-449. Forget, R. and Y. Lamarre (1987). "Rapid elbow flexion in the absence of proprioceptive and cutaneous feedback." Human Neurobiology 6: 27-37. Franklin, D., E. Burdet, et al. (2003). "Functional significance of stiffness in adaptation of multijoint arm movements to stable and unstable dynamics." Experimental Brain Research 151(2): 145-157. 294 Franklin, D. W., E. Burdet, et al. (2008). "CNS Learns Stable, Accurate, and Efficient Movements Using a Simple Algorithm." The Journal of Neuroscience 28(44): 11165-11173. Franklin, D. W., R. Osu, et al. (2003). "Adaptation to Stable and Unstable Dynamics Achieved By Combined Impedance Control and Inverse Dynamics Model." Journal of Neurophysiology 90(5): 3270-3282. Franklin, D. W., U. So, et al. (2004). "Impedance control balances stability with metabolically costly muscle activation." Journal of Neurophysiology 92: 3097-3105. Fuglevand, A. J., D. A. Winter, et al. (1993). "Models of recruitment and rate coding organization in motor-unit pools." J.Neurophysiol 70(6): 2470-2488. Gandolfo, F., F. A. Mussa-Ivaldi, et al. (1996). "Motor learning by field approximation." Proc.Natl.Acad.Sci.U.S.A 93(9): 3843-3846. Ganesh, G., M. Haruno, et al. (2010). "Motor Memory and Local Minimization of Error and Effort, Not Global Optimization, Determine Motor Behavior." Journal of Neurophysiology 104(1): 382-390. Gelfan, S. (1964). Neuronal Interdependence. Progress in Brain Research. J. C. Eccles and J. P. Schadé, Elsevier. Volume 11: 238-260. Ghez, C. and J. Gordon (1987). "Trajectory control in targeted force impulses I. Role of opposing muscles." Experimental Brain Research 67: 225-240. Ghez, C. and J. H. Martin (1982). "The control of rapid limb movement in the cat. III. Agonist-antagonist coupling." Experimental Brain Research 45: 115-125. Ghez, C. and R. Sainburg (1994). Proprioceptive control of interjoint coordination. Gielen, S., G. Schenau, et al. (1990). The activation of mono-and bi-articular muscles in multi-joint movements. Multiple muscle systems: Biomechanics and movement. J. Winters and S. Woo. New York, Springer-Verlag: 303-311. González-Alonso, J., B. Quistorff, et al. (2000). "Heat production in human skeletal muscle at the onset of intense dynamic exercise." The Journal of Physiology 524(2): 603-615. Goodner, J., G. Tsianos, et al. (2011). "A physiologically plausible learning model for the sensorimotor system." Soc Neurosci Abstr 387.02. 295 Gordon, A. M., E. Homsher, et al. (2000). "Regulation of contraction in striated muscle." Physiological Reviews 80(2): 853-924. Gordon, J., M. Iyer, et al. (1987). "Impairment of motor programming and trajectory control in a deafferented patient." Soc Neurosci Abstr 13: 352. Gracies, J. M., S. Meunier, et al. (1991). "Pattern of propriospinal-like excitation to different species of human upper limb motoneurons." Journal of Physiology 434: 156-167. Graham, K. M., K. D. Moore, et al. (2003). "Kinematics and kinetics of multijoint reaching in nonhuman primates." J.Neurophysiol. 89(5): 2667-2677. Grashow, R., T. Brookings, et al. (2010). "Compensation for Variable Intrinsic Neuronal Excitability by Circuit-Synaptic Interactions." The Journal of Neuroscience 30(27): 9145-9156. Gregory, J. E. and U. Proske (1979). "The responses of Golgi tendon organs to stimulation of different combinations of motor units." The Journal of Physiology 295(1): 251-262. Gribble, P. L., L. I. Mullin, et al. (2003). "Role of cocontraction in arm movement accuracy." J.Neurophysiol. 89: 2396-2405. Hallett, M., B. T. Shahani, et al. (1975). "EMG analysis of stereotyped voluntary movements in man." Journal of Neurology, Neurosurgery, and Psychiatry 38: 1154-1162. Hasan, Z. (2005). "The human motor control system's response to mechanical perturbation: should it, can it, and does it ensure stability?" Journal of Motor Behavior 37(6): 484-493. Hasan, Z. and G. M. Karst (1989). "Muscle activity for initiation of planar, two-joint arm movements in different directions." Exp.Brain Res. 76(3): 651-655. Hatze, H. and J. D. Buys (1977). "Energy-optimal controls in the mammalian neuromuscular system." Biol.Cybernetics 27: 9-27. He, J., W. S. Levine, et al. (1991). "Feedback gains for correcting small perturbations to standing posture." IEEE Transactions on Automatic Control 36(3): 322- 332. Henneman, E. and L. M. Mendell (1981). Functional organization of the motoneuron pool and its inputs. Handbook of Physiology. Sect. I. The Nervous System. Vol 296 II, Part 1. V. B. Brooks. Washington, DC, American Physiological Society: 423- 507. Hoffer, J. A., A. A. Caputi, et al. (1989). "Roles of muscle activity and load on the relationship between muscle spindle length and whole muscle length in the freely walking cat." Prog.Brain Res. 80: 75-85. Hoffer, J. A., N. Sugano, et al. (1987). "Cat hindlimb motoneurons during locomotion. II. Normal activity patterns." J.Neurophysiol. 57(2): 530-553. Hogan, N., E. Bizzi, et al. (1987). "Controlling multijoint motor behavior." Exerc.Sport Sci.Rev. 15: 153-190. Hollerbach, J. M. and T. Flash (1982). "Dynamic interactions between limb segments during planar arm movement." Biological Cybernetics 44: 67-77. Hongo, T., S. Kitazawa, et al. (1989). "A physiological and morphological study of premotor interneurones in the cutaneous reflex pathways in cats." Brain Research 505: 163-166. Hongo, T., S. Kitazawa, et al. (1989). "Functional identification of last-order interneurones of skin reflex pathways in the cat forelimb segments." Brain Research 505: 167-170. Houdijk, H., M. F. Bobbert, et al. (2006). "Evaluation of a Hill based muscle model for the energy cost and efficiency of muscular contraction." Journal of Biomechanics 39(3): 536-543. Houk, J. and E. Henneman (1967). "Responses of Golgi tendon organs to active contractions of the soleus muscle of the cat." Journal of Neurophysiology 30(466): 481. Houk, J. C. (1979). "Regulation of stiffness by skeletomotor reflexes." Annual Review of Physiology 41: 99-114. Hoy, M. G., F. E. Zajac, et al. (1990). "A musculoskeletal model of the human lower extremity: the effect of muscle, tendon, and moment arm on the moment- angle relationship of musculotendon actuators at the hip, knee, and ankle." Journal of Biomechanics 23: 157-169. Hultborn, H., M. Illert, et al. (1976). "Convergence on Interneurones Mediating the Reciprocal Ia Inhibition of Motoneurones I. Disynaptic Ia Inhibition of Ia Inhibitory Interneurones." Acta Physiologica Scandinavica 96(2): 193-201. 297 Hultborn, H., E. Jankowska, et al. (1971). "Recurrent inhibition of interneurones monosynaptically activated from group Ia afferents." The Journal of Physiology 215(3): 613-636. Hultborn, H., E. Pierrot-Deseilligny, et al. (1979). "Recurrent inhibition and afterhyperpolarization following motoneuronal discharge in the cat." The Journal of Physiology 297(1): 253-266. Illert, M. and H. Kummel (1999). "Reflex pathways from large muscle spindle afferents and recurrent axon collaterals to motoneurones of wrist and digit muscles: a comparison in cats, monkeys and humans." Experimental Brain Research 128: 13-19. Illert, M., A. Lundberg, et al. (1975). "Convergence on propriospinal neurones which may mediate disynaptic conticospinal excitation to forelimb motoneurones in the cat." Brain Research 93: 530-534. Illert, M., A. Lundberg, et al. (1978). "Integration in descending motor pathways controlling the forelimb in the cat." Experimental Brain Research 33(1): 101- 130. Isa, T., Y. Ohki, et al. (2007). "Direct and indirect cortico-motoneuronal pathways and control of hand/arm movements." Physiology 22: 145-152. Ivashko, D. G., B. I. Prilutsky, et al. (2003). "Modeling the spinal cord neural circuitry controlling cat hindlimb movement during locomotion." Neurocomputing 52- 54: 621-629. Jankowska, E. (1992). "Interneuronal relay in spinal pathways from proprioceptors." Progress in Neurobiology 38: 335-378. Jankowska, E. (2008). "Spinal interneuronal networks in the cat: elementary components." Brain Research Reviews 57: 46-55. Jankowska, E. and S. Lindstrom (1971). "Morphological identification of Renshaw cells." Acta Physiol Scand 81: 428-430. Jankowska, E. and S. Lindström (1972). "Morphology of interneurones mediating Ia reciprocal inhibition of motoneurones in the spinal cord of the cat." The Journal of Physiology 226(3): 805-823. Jessell, T. M. (2000). "Neuronal specification in the spinal cord: inductive signals and transcriptional codes." Nat Rev Genet 1(1): 20-29. 298 Jones, K. E., A. F. d. Hamilton, et al. (2002). "Sources of signal-dependent noise during isometric force production." Journal of Neurophysiology 88: 1533- 1544. Jones, L. A. and I. W. Hunter (1983). "Effect of fatigue on force sensation." Experimental Neurology 81(3): 640-650. Joyce, G. S., P. M. H. Rack, et al. (1969). "Mechanical properties of cat soleus muscle during controlled lengthening and shortening movements." Journal of Physiology (London) 204: 461-474. Kautz, S. A., M. L. Hull, et al. (1994). "A comparison of muscular mechanical energy expenditure and internal work in cycling." Journal of Biomechanics 27(12): 1459-1467. Keenan, K. G. and F. J. Valero-Cuevas (2007). "Neural mechanisms dominate the experimentally-valid predictions of muscle force and EMG in models of motor unit function." Submitted to Journal of Neurophysiology May 22, 2007. Kelso, T. B., D. R. Hodgson, et al. (1987). "Some properties of different skeletal muscle fiber types: comparison of reference bases." Journal of Applied Physiology 62(4): 1436-1441. Kitazawa, S., Y. Ohki, et al. (1993). "Candidate premotor neurones of skin reflex pathways to T1 forelimb motoneurones of the cat." Experimental Brain Research 95(2): 291-307. Kluzik, J., J. Diedrichsen, et al. (2008). "Reach Adaptation: What Determines Whether We Learn an Internal Model of the Tool or Adapt the Model of Our Arm?" Journal of Neurophysiology 100: 1455-1464. Kumamoto, M., T. Oshima, et al. (1994). "Control properties induced by the existence of antagonistic pairs of bi-articular muscles - Mechanical engineering model analyses." Human Movement Science 13: 611-634. Lacquaniti, F. and J. F. Soechting (1986). "Responses of mono- and bi-articular muscles to load perturbations of the human arm." Experimental Brain Research 65: 135-144. Lan, N., Y. Li, et al. (2005). "Reflex regulation of antagonist muscles for control of joint equilibrium position." IEEE Transactions on Neural Systems and Rehabilitation Engineering 13(1): 60-71. 299 Lavoie, S., B. McFadyen, et al. (1995). "A kinematic and kinetic analysis of locomotion during voluntary gait modification in the cat." Experimental Brain Research 106: 39-56. Lee, R. G. and W. G. Tatton (1975). "Motor responses to sudden limb displacements in primates with specific CNS lesions and in human patients with motor system disorders." Canadian Journal of Neurological Sciences 2(3): 285-293. Lee, S.-K. and S. L. Pfaff (2001). "Transcriptional networks regulating neuronal identity in the developing spinal cord." Nat Neurosci 4: 1183 - 1191 Leijendekker, W. J. and G. Elzinga (1990). "Metabolic recovery of mouse extensor digitorum longus and soleus muscle." Pflügers Archiv European Journal of Physiology 416(1): 22-27. Lieber, R. L. (1990). "Hypothesis: biarticular muscles transfer moments between joints." Dev.Med.Child Neurol. 32(5): 456-458. Lin, C.-C. K. and P. E. Crago (2002). "Structural Model of the Muscle Spindle." Annals of Biomedical Engineering 30(1): 68-83. Linari, M., R. C. Woledge, et al. (2003). "Energy storage during stretch of active single fibres from frog skeletal muscle." The Journal of Physiology 548(2): 461-474. Lloyd, D. P. C. (1946). "Integrative pattern of excitation and inhibition in two-neuron reflex arcs." Journal of Neurophysiology 9: 439-444. Loeb, G. E. (1993). "The distal hindlimb musculature of the cat: interanimal variability of locomotor activity and cutaneous reflexes." Exp.Brain Res. 96(1): 125-140. Loeb, G. E. (2012). "Optimal isn’t good enough." Biological Cybernetics: 1-9. Loeb, G. E., I. E. Brown, et al. (1999). "A hierarchical foundation for models of sensorimotor control." Exp.Brain Res. 126(1): 1-18. Loeb, G. E., I. E. Brown, et al. (2002). "The importance of biomechanics." Adv.Exp.Med.Biol. 508: 481-487. Loeb, G. E., I. E. Brown, et al. (1996). "Directional motor control." Trends in Neurosciences 19: 137-138. Loeb, G. E., J. He, et al. (1989). "Spinal cord circuits: are they mirrors of musculoskeletal mechanics?" Journal of Motor Behavior 21: 473-491. 300 Loeb, G. E., W. S. Levine, et al. (1990). "Understanding sensorimotor feedback through optimal control." Cold Spring Harbor Symposia on Quantitative Biology 55: 791-803. Loeb, G. E. and F. J. R. Richmond (1989). "Motor partitioning: epiphenomena masquerading as control theory." Behavioral and Brain Sciences 12: 660-661. Luff, A. R. and H. L. Atwood (1971). "Changes in the sarcoplasmic reticulum and transverse tubular system of fast and slow skeletal muscles of the mouse during postnatal development." The Journal of Cell Biology 51(2): 369-383. Lundberg, A., K. Malmgren, et al. (1987). "Reflex pathways from group II muscle afferents. 3. Secondary spindle afferents and the FRA: a new hypothesis." Experimental Brain Research 65: 294-306. Lyndon, H. (1989). " I did it my way! An introduction to “old way / new way”methodology " Australasian J Special Education 13: 32-37. Ma, S. and G. I. Zahalak (1991). "A distribution-moment model of energetics in skeletal muscle." J Biomech 24(1): 21-35. Magid, A. and D. J. Law (1985). "Myofibrils bear most of the resting tension in frog skeletal muscle." Science 230: 1280-1282. Maier, M. A., S. I. Perlmutter, et al. (1998). "Response patterns and force relations of monkey spinal interneurons during active wrist movement." Journal of Neurophysiology 80: 2495-2513. Maier, M. A., L. E. Shupe, et al. (2005). "Dynamic Neural Network Models of the Premotoneuronal Circuitry Controlling Wrist Movements in Primates." Journal of Computational Neuroscience 19: 125-146. Malamud, J. G., R. E. Godt, et al. (1996). "Relationship between short-range stiffness and yielding in type-identified, chemically skinned muscle fibers from the cat triceps surae muscles." Journal of Neurophysiology 76(4): 2280-2289. Malmgren, K. and E. Pierrot-Deseilligny (1988). "Evidence for non-monosynaptic Ia excitation of human wrist flexor motoneurones, possibly via propriospinal neurones." Journal of Physiology 405: 747-764. Malmgren, K. and E. Pierrot-Deseilligny (1988). "Inhibition of neurones transmitting nonmonosynaptic Ia excitation to human wrist flexor motoneurones." Journal of Physiology 405: 765-783. 301 Maltenfort, M. G. and R. E. Burke (2003). "Spindle Model Responsive to Mixed Fusimotor Inputs and Testable Predictions of β Feedback Effects." Journal of Neurophysiology 89(5): 2797-2809. Massion, J. (1992). "Movement, posture and equilibrium: Interaction and coordination." Progress in Neurobiology 38: 35-56. Matise, M. P. and A. L. Joyner (1997). "Expression Patterns of Developmental Control Genes in Normal andEngrailed-1 Mutant Mouse Spinal Cord Reveal Early Diversity in Developing Interneurons." The Journal of Neuroscience 17(20): 7805-7816. Mattar, A. A. G. and D. J. Ostry (2007). "Modifiability of Generalization in Dynamics Learning." Journal of Neurophysiology 98(6): 3321-3329. Mattar, A. A. G. and D. J. Ostry (2010). "Generalization of Dynamics Learning Across Changes in Movement Amplitude." Journal of Neurophysiology 104(1): 426- 438. Maynard, E. M., N. G. Hatsopoulos, et al. (1999). "Neuronal interactions improve cortical population coding of movement direction." J Neurosci. 19(18): 8083- 8093. McCrea, D. A. (1986). "Spinal cord circuitry and motor reflexes." Exercise and Sport Sciences Reviews 14: 105-141. McCrea, D. A. and I. A. Rybak (2008). "Organization of mammalian locomotor rhythm and pattern generation." Brain Research Reviews 57(1): 134-146. Meunier, S., E. Pierrot-Deseilligny, et al. (1993). "Pattern of monosynaptic heteronymous Ia connections in the human lower limb." Experimental Brain Research 96: 534-544. Mileusnic, M. P., I. E. Brown, et al. (2006). "Mathematical models of proprioceptors. I. Control and transduction in the muscle spindle." J.Neurophysiol. 96(4): 1772-1788. Mileusnic, M. P. and G. E. Loeb (2009). "Force estimation from ensembles of Golgi tendon organs." J.Neural Eng 6: 1-15. Milner, T. E. and D. W. Franklin (2005). "Impedance control and internal model use during the initial stage of adaptation to novel dynamics in humans." The Journal of Physiology 567(2): 651-664. 302 Neptune, R. R. and A. J. Van Den Bogert (1998). "Standard mechanical energy analyses do not correlate with muscle work in cycling." Journal of Biomechanics 31(3): 239-245. O'Connor, J. J. (1993). "Can muscle co-contraction protect knee ligaments after injury or repair?" Journal of Bone and Joint Surgery - British Volume 75-B(1): 41-48. Ostry, D. J. and A. G. Feldman (2003). "A critical evaluation of the force control hypothesis in motor control." Experimental Brain Research 221: 275-288. Osu, R. and H. Gomi (1999). "Multijoint muscle regulation mechanisms examined by measured human arm stiffness and EMG signals." Journal of Neurophysiology 81: 1458-1468. Pandy, M. G. and F. E. Zajac (1991). "Optimal muscular coordination strategies for jumping." J Biomech. 24(1): 1-10. Pedotti, A., V. V. Krishnan, et al. (1978). "Optimization of muscle-force sequencing in human locomotion." Mathematics and Bioscience 38: 57-76. Perlmutter, S. I., M. A. Maier, et al. (1998). "Activity of spinal internerons and their effects on forearm muscles during voluntary wrist movements in the monkey." Journal of Neurophysiology 80: 2475-2494. Perreault, E. J., C. J. Heckman, et al. (2003). "Hill muscle model errors during movement are greatest within the physiologically relevant range of motor unit firing rates." Journal of Biomechanics 36(2): 211-218. Pierrot-Deseilligny, E. and D. C. Burke (2005). The Circuitry of the Human Spinal Cord: Its Role in Motor Control and Movement Disorders, Cambridge University Press. Illustrated. Pierrot-Deseilligny, E., C. Morin, et al. (1981). "Pattern of group I fibre projections from ankle flexor and extensor muscles in man." Experimental Brain Research 42(3): 337-350. Pierrynowski, M. R. and J. B. Morrison (1985). "Estimating the muscle forces generated in the human lower extremity when walking: a physiological solution." Mathematics and Bioscience 75: 43-68. Pincivero, D. M., Y. Salfetnikov, et al. (2004). "Angle- and gender-specific quadriceps femoris muscle recruitment and knee extensor torque." Journal of Biomechanics 37(11): 1689-1697. 303 Pinniger, G. J., K. W. Ranatunga, et al. (2006). "Crossbridge and non-crossbridge contributions to tension in lengthening rat muscle: force-induced reversal of the power stroke." The Journal of Physiology 573(3): 627-643. Ramon y Cajal, S. (1907). "Die struktur der sensiblen ganglien des menschen und der tiere." Ergebn d Anat u Entwicklungsgesch 16: 177-217. Raphael, G., G. A. Tsianos, et al. (2010). "Spinal-Like Regulator Facilitates Control of aTwo-Degree-of-Freedom Wrist." The Journal of Neuroscience 30(28): 9431- 9444. Rathelot, J. A. and P. L. Strick (2009). "Subdivisions of primary motor cortex based on cortico-motoneuronal cells." PNAS 106(3): 918-923. Ritter, D. A., D. H. Bhatt, et al. (2001). "In Vivo Imaging of Zebrafish Reveals Differences in the Spinal Networks for Escape and Swimming Movements." The Journal of Neuroscience 21(22): 8956-8965. Rothwell, J. C., M. M. Traub, et al. (1982). "Manual motor performance in a deafferented man." Brain 105: 515-542. Ryall, R. W. (1981). "Patterns of recurrent excitation and mutual inhibition of cat Renshaw cells." The Journal of Physiology 316(1): 439-452. Sanes, J. N. and V. A. Jennings (1984). "Centrally Programmed Patterns of Muscle Activity in Voluntary Motor Behavior of Humans." Experimental Brain Research 54: 23-32. Saugen, E. and N. K. Vollestad (1995). "Nonlinear relationship between heat production and force during voluntary contractions in humans." Journal of Applied Physiology 79(6): 2043-2049. Schouten, A. C., E. de Vlugt, et al. (2001). "Optimal posture control of a musculo- skeletal arm model." Biol.Cybern. 84(2): 143-152. Schutte, L. (1992). Using musculoskeletal models to explore strategies for improving performance in electrical stimulation-induced leg cycle ergometry, Stanford University. Scott, S. H. (2000). "Population vectors and motor cortex: neural coding or epiphenomenon?" Nat.Neurosci. 3(4): 307-308. Scott, S. H., I. E. Brown, et al. (1996). "Mechanics of feline soleus: I. Effect of fascicle length and velocity on force output." J.Muscle Res.Cell Motil. 17: 205-218. 304 Scott, S. H. and J. F. Kalaska (1995). "Changes in motor cortex activity during reaching movements with similar hand paths but different arm postures." J.Neurophysiol. 73(6): 2563-2567. Scott, S. H. and G. E. Loeb (1995). "Mechanical properties of aponeurosis and tendon of the cat soleus muscle during whole-muscle isometric contractions." J.Morphol. 224(1): 73-86. Shadmehr, R. and F. A. Mussa-Ivaldi (1994). "Adaptive representation of dynamics during learning of a motor task." J.Neurosci. 14(5 Pt 2): 3208-3224. Shelburne, K. B. and M. G. Pandy (1997). "A musculoskeletal model of the knee for evaluating ligament forces during isometric contractions." Journal of Biomechanics 30(2): 163-176. Sherrington, C. S. (1906). The integrative action of the nervous system. New Haven, London, Yale University Press. Smith, N. P., C. J. Barclay, et al. (2005). "The efficiency of muscle contraction." Progress in Biophysics and Molecular Biology 88: 1-58. Somlyo, A. V., H. G. Gonzalez-Serratos, et al. (1981). "Calcium release and ionic changes in the sarcoplasmic reticulum of tetanized muscle: an electron-probe study." The Journal of Cell Biology 90(3): 577-594. Song, D., P. Hendrickson, et al. (2008). Predicting EMG with generalized Volterra kernel model. Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE. Song, D., G. Raphael, et al. (2008). "Computationally efficient models of neuromuscular recruitment and mechanics." Journal of Neural Engineering 5: 175-184. Stein, P. S. G. (1978). "Motor systems, with specific reference to the control of locomotion." Annual Review of Neuroscience 1: 61-81. Stracke, C. P., L. G. Pettersson, et al. (2005). "Interneuronal systems of the cervical spinal cord assessed with BOLD imaging at 1.5 T." Neuroradiology 47(2): 127-133. Sunwoo, J., G. A. Tsianos, et al. (2012). "Improving the Dynamic Programming of a Spinal-Like Regulator by Using Preceding Results to Devise Efficient Search Strategies." BMES. 305 Tanaka, R. (1974). "Reciprocal Ia inhibition during voluntary movements in man." Experimental Brain Research 21(5): 529-540. Thoroughman, K. A. and R. Shadmehr (1999). "Electromyographic Correlates of Learning an Internal Model of Reaching Movements." The Journal of Neuroscience 19(19): 8573-8588. Todorov, E. and M. I. Jordan (2002). "Optimal feedback control as a theory of motor coordination." Nature Neuroscience 5(11): 1226-1235. Trinh, M. and D. A. Syme (2007). "Effects of stretch on work and efficiency of frog (Rana pipiens) muscle." Journal of Experimental Biology 210(16): 2843- 2850. Tsianos, G. A., G. Raphael, et al. (2011). "Modeling the potentiality of spinal-like circuitry for stabilization of a planar arm system." Prog Brain Res 194: 203- 213. Tsianos, G. A., C. Rustin, et al. (2012). "Mammalian muscle model for predicting force and energetics during physiological behaviors." IEEE Trans Neural Syst Rehabil Eng 20(2): 117-133. Umberger, B. R., K. G. M. Gerritsen, et al. (2003). "A model of human muscle energy expenditure." Computer Methods in Biomechanics and Biomedical Engineering 6(2): 99-111. Valero-Cuevas, F. J., H. Hoffmann, et al. (2009). "Computational models for neuromuscular function." IEEE Reviews in Biomedical Engineering 2: 110- 135. van Heijst, J. J., J. E. Vos, et al. (1998). "Development in a biologically inspired spinal neural network for movement control." Neural Networks 11(7-8): 1305- 1316. Van Ingen Schenau, G. J. (1989). "From rotation to translation: constraints on multi- joint movements and the unique action of bi-articular muscles." Human Movement Science 8: 301-337. Weber, D. J. and J. He (2004). "Adaptive behavior of cortical neurons during a perturbed arm-reaching movement in a nonhuman primate." Prog.Brain Res. 143: 477-490. Wendt, I. R. and C. L. Gibbs (1976). "Recovery heat production of mammalian fast- and slow-twitch muscles." American Journal of Physiology 230(6): 1637- 1643. 306 Westerblad, H., J. D. Bruton, et al. (2010). "Skeletal muscle: Energy metabolism, fiber types, fatigue and adaptability." Experimental Cell Research: 3093-3099. Westwick, D. T., E. A. Pohlmeyer, et al. (2006). "Identification of Multiple-Input Systems with Highly Coupled Inputs: Application to EMG Prediction from Multiple Intracortical Electrodes." Neural Computation 18(2): 329-355. Woledge, R. C., C. J. Barclay, et al. (2009). "Temperature change as a probe of muscle crossbridge kinetics: a review and discussion." Proc R Soc B 276: 2685-2695. Woledge, R. C., N. A. Curtin, et al. (1985). Energetic aspects of muscle contraction London, Academic Press. Yakovenko, S., N. Krouchev, et al. (2011). "Sequential Activation of Motor Cortical Neurons Contributes to Intralimb Coordination During Reaching in the Cat by Modulating Muscle Synergies." Journal of Neurophysiology 105(1): 388-409. Zahalak, G. I. (1981). "A Distribution-Moment Approximation for Kinetic Theories of Muscular Contraction." Mathematical Biosciences 55: 89-114. Zajac, F. E. (1985). "Thigh muscle activity during maximum-height jumps by cats." J Neurophysiol. 53(4): 979-994. Zajac, F. E. (1989). "Muscle and tendon: properties, models, scaling and application to biomechanics and motor control." Critical Reviews in Biomedical Engineering 17: 359-411. Zajac, F. E., M. R. Zomlefer, et al. (1981). "Hindlimb muscular activity, kinetics and kinematics of cats jumping to their maximum achievable heights." Journal of Experimental Biology 91: 73-86.
Abstract (if available)
Abstract
Making voluntary movements requires proper recruitment of muscles that exert torques at the joints. The brain controls these torques indirectly by sending commands to the spinal circuitry, which continuously integrates them with proprioceptive feedback and recurrent projections from motoneurons. The nature of this transformation and its implications for motor control have been investigated by building a realistic model of the lower motor system (spinal circuitry plus musculoskeletal system) and determining how much of the dynamics of reaching movement it can generate entirely on its own. ❧ An oversimplified model of the brain was employed in order to force the lower motor system to generate all of the necessary dynamics. Its outputs, representing supraspinal control of fusimotor gain, interneuronal biasing activity and presynaptic inhibition/facilitation, controlled a realistic set of spinal circuits based on the classical interneuronal types (propriospinal, monosynaptic Ia-excitatory, reciprocal Ia-inhibitory, Renshaw inhibitory and Ib-inhibitory pathways). Commands to the spinal circuitry were unmodulated step functions whose amplitudes were trained using a simple optimization algorithm and a cost function. ❧ Despite the large number of control points in the spinal cord model (greater than 400 for our six-muscle model) and the oversimplified descending inputs, it was surprisingly easy to train the system to perform motor tasks such as resisting an impulsive perturbation applied to the endpoint and center-out reaches to multiple directions along with the complex muscle dynamics required to achieve them. This is because the high-dimensional space to be controlled appears to have many ""good enough"" solutions and relatively few undesirable local minima. Initially, energetics were not part of the performance criteria
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Spinal-like regulator for control of multiple degree-of-freedom limbs
PDF
Properties of human motor control under risk and risk aware control
PDF
Model-based studies of control strategies for noisy, redundant musculoskeletal systems
PDF
On the electrophysiology of multielectrode recordings of the basal ganglia and thalamus to improve DBS therapy for children with secondary dystonia
PDF
A non-invasive and intuitive command source for upper limb prostheses
PDF
Experimental and model-based analyses of physiological determinants of force variability
PDF
Design and use of a biomimetic tactile microvibration sensor with human-like sensitivity and its application in texture discrimination using Bayesian exploration
PDF
insideOut: Estimating joint angles in tendon-driven robots using Artificial Neural Networks and non-collocated sensors
PDF
Finite element model to understand the role of fingerprints in generating vibrations
PDF
A percutaneously implantable wireless neurostimulator for treatment of stress urinary incontinence
PDF
Iterative path integral stochastic optimal control: theory and applications to motor control
PDF
Understanding the pathology of dystonia by hardware emulation
PDF
Bio-inspired tendon-driven systems: computational analysis, optimization, and hardware implementation
PDF
Electromyography of spinal cord injured rodents trained by neuromuscular electrical stimulation timed to robotic treadmill training
PDF
Deficits and rehabilitation of upper-extremity multi-joint movements in individuals with chronic stroke
PDF
Neurophysiological bases of the strength‐dexterity paradigm with the use of H‐reflex
PDF
Neuromuscular dynamics in the context of motor redundancy
PDF
Inference of computational models of tendon networks via sparse experimentation
PDF
Task-dependent modulation of corticomuscular coherence during dexterous manipulation
PDF
The representation, learning, and control of dexterous motor skills in humans and humanoid robots
Asset Metadata
Creator
Tsianos, George A.
(author)
Core Title
Investigating the role of muscle physiology and spinal circuitry in sensorimotor control
School
Viterbi School of Engineering
Degree
Doctor of Philosophy
Degree Program
Biomedical Engineering
Publication Date
10/10/2012
Defense Date
09/04/2012
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
motor control,motor memory,muscle,OAI-PMH Harvest,reaching,spinal circuitry
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Loeb, Gerald E. (
committee chair
), Marmarelis, Vasilis Z. (
committee member
), Sanger, Terence D. (
committee member
), Schaal, Stefan (
committee member
), Valero-Cuevas, Francisco J. (
committee member
)
Creator Email
george.a.tsianos@gmail.com,tsianos@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c3-104110
Unique identifier
UC11290244
Identifier
usctheses-c3-104110 (legacy record id)
Legacy Identifier
etd-TsianosGeo-1242.pdf
Dmrecord
104110
Document Type
Dissertation
Rights
Tsianos, George A.
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
motor control
motor memory
muscle
reaching
spinal circuitry