Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Mechanical and flow interactions facilitate cooperative transport and collective locomotion in animal groups
(USC Thesis Other)
Mechanical and flow interactions facilitate cooperative transport and collective locomotion in animal groups
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
Mechanical and flow interactions facilitate cooperative transport and collective locomotion in animal groups by Sina Heydari A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (MECHANICAL ENGINEERING) May 2023 Copyright 2023 Sina Heydari Dedication To my loved ones and their unwavering support. ii Acknowledgments The completion of this PhD was only made possible by the support and help of many wonderful people in my life. I would like to take this opportunity to express my gratitude to some of them. Firstly, I would like to thank my advisor Prof. Eva Kanso for her guidance and support throughout my PhD. I’ve learned many precious lessons from her that I could apply to my professional and personal life. I want to thank her her trust in me and for allowing me undertake this journey under her supervision. She has been an exemplary role model for me in work ethic, motivation and research curiosity. Above all, she has taught me the value of being persistent and consisten in my professional life. A lesson that I hope to carry with me into my future career. I would also like to express my gratitude to Prof. Matthew J. McHenry for his invaluable collaboration and insight. As an engineer, I am always fascinated by the depth of his knowledge in biology and his ability to formulate and perform intricate experimental studies. The first half of this dissertation would not have been possible, if it were not for the hours of discussion we had. I also want to thank him for letting me visit his lab and watch the locomotion of sea stars in person! I am deeply grateful to my other committee members Prof. Mitul Luhar and Prof. Aiichiro Nakano. Not only for their constructive feedback on my qualifying exam proposal, but also for their fun and stimulating classes. My PhD research is built upon what I’ve learned about fluid dynamics and computational physics in their classes. In addition, I would like to thank Prof. Luhar for his collaboration on the Starbot project. iii Secondly, I want to acknowledge the great friendships that I have made during my time at USC. I want to thank Basile, Feng, Anup, Yusheng, Jingyi, Chenchen, Haotian, Morgan and Yi for their welcom- ing presence and the fun memories we made. I will always look back at our rock climbing sessions and backpacking trip with fondness. I want to especially acknowledge Haotian for being such a hardworking colleague and for putting so much effort into our collaborative work (parts of which are included in this dissertation). I also want to thank Nami for being my first and best friend in Los Angeles. Last but not least, I would like to express my utmost gratitude to my family. Especially my parents. I am forever indebted to them for their support and for raising me into the person I am today. I would like to thank my sister, Bita, for always providing me with emotional support and advice when I needed it. I want to thank my brother-in-law, Ben, for being the brother I never had. And I want to thank my dear nephew, Dion, for making my last year of PhD so much more colorful with his presence. Above all , I am thankful to my lovely wife, Nazanin, for always being there for me. Her unconditional love and companionship is what got me through the most difficult times in my PhD. iv TableofContents Dedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi I Cooperativetransportusingtubefeetinspiredactuators 1 Chapter 1: Introduction and motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Chapter 2: Tube feet biomechanics and model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1 Tube feet as soft hydrostatic skeletons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Mathematical model of tube-feet inspired actuators . . . . . . . . . . . . . . . . . . . . . . 7 Chapter 3: Tube feet inspired locomotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.1 Sea stars body mechanics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2 Hierarchical structure for tube feet control . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.3 Point mass carried by two tube feet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.4 Locomotion using an array of ten tube feet . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Chapter 4: Learning optimal mechanosensory cues for cooperative locomotion . . . . . . . . . . . 30 4.1 Tube feet control via Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.2 Learning the control policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.3 Testing the control policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.4 Decentralized control with local directionality command . . . . . . . . . . . . . . . . . . . 38 4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 II Self-organizationofflow-coupledswimmersintospatialpatterns 41 Chapter 5: Introduction and motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Chapter 6: Vortex sheet model of flapping swimmers . . . . . . . . . . . . . . . . . . . . . . . . . . 45 v 6.1 Flow field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 6.2 Forces and moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 6.3 Numerical implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Chapter 7: Self-organization in pairs of flapping swimmers . . . . . . . . . . . . . . . . . . . . . . 53 7.1 Mathematical modeling of flow-coupled swimmers . . . . . . . . . . . . . . . . . . . . . . 53 7.2 Universal spacing-phase relationship in emergent pairwise formations . . . . . . . . . . . 54 7.3 Insights into power savings and fluid forces from tethered formations . . . . . . . . . . . . 58 7.4 Parametric analysis over the entire space of phase lags and lateral offsets . . . . . . . . . . 60 7.5 Opportunities in the wake of a solitary swimmer . . . . . . . . . . . . . . . . . . . . . . . . 62 7.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Chapter 8: Emergent formations in large groups of swimmers . . . . . . . . . . . . . . . . . . . . . 68 8.1 Stability of large groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 8.2 Cooperative and selfish energy savings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 8.3 The effect of flow parameters on emergent formations . . . . . . . . . . . . . . . . . . . . . 77 8.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Chapter 9: Conclusion and outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Appendices 96 Appendix A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Cooperative transport using tube feet inspired actuators . . . . . . . . . . . . . . . . . . . . . . . 97 A.1 Reinforcement Learning Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Appendix B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Self-organization of flow-coupled swimmers into spatial patterns . . . . . . . . . . . . . . . . . . 103 B.1 CFD model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 B.2 Time-delay particle model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 B.3 Hydrodynamic power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 B.4 Stability analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 vi ListofTables 3.1 Sea star parameters (based on [116, 64, 2]) . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.1 Candidate mechanosensory cues for control submodalities. . . . . . . . . . . . . . . . . . 34 8.1 Average power saving of groups of swimmers shown in Figs. 8.4, 8.5, 8.6 relative to a solitary swimmer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 8.2 Swimming speed and hydrodynamic power of a solitary swimmer and equilibrium separation distance and average power saving in a pair of swimmers for different dissipation timesT diss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 vii ListofFigures 2.1 A. The common sea star Asterias rubens (source: Shutterstock),B. close-up on the tube feet lining the ventral surface of Asterias rubens (source: Symbiotic service, San Diego),C. tube foot anatomy for an adult sea star, D. Activation of podia and ampulla muscles lead to contraction and extension of the tube feet. E. tube foot anatomy for an adult sea star, F. muscles are innervated by neurons located in the radial nerves and nerve ring. Activation of podia and ampulla muscles lead to contraction, extension and bending of the tube feet. . . 7 2.2 Tube foot inspired soft actuator. A. when attached to a substrate, a tube foot generates either a pushing or a pulling force on the body it is attached to. The active force profiles (orange lines) are inspired by Hill’s muscle model (grey lines). Vertical motion of B. a point mass carried by a single vertical actuator andC. a rigid body of the same mass carried by an array of 10 identical actuators. . . . . . . . . . . . . . . . . . . . . . . 9 3.1 Mathematical model of locomotion. A. schematic of our mechanical model of the sea star and tube feet inspired actuators, with inset showing contractile, passive and dissipative force elements along each tube foot,B. hierarchical motor control of the tube feet consisting of global directionality commands issued by the radial nerves and nerve ring and local sensory-motor feedback loops at the tube foot level. . . . . . . . . . 16 3.2 Bipedal locomotion. A point mass attached to two tube feet, each producing longitudinal pushing or pulling forces along the direction of the foot in the attached phase (power stroke) and taking a step forward in the detached phase (recovery stroke). A. Trajectory of the point mass in the(x,y) plane and snapshots of the walker at three instants in time. B. Orientation angles of the tube feet versus time. C. Active forces along the tube feet as a function of time. Positive force corresponds to pushing and negative force corresponds to pulling. The parameter values are set toFmax = 2, mg = 1 andγ = 10. The step size taken after a detachment-reattachment cycle is∆ θ =π/ 6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.3 Crawl and bounce gaits: of a sea star model with ten tube feet. A. Trajectory of the sea star center of mass in the(x,y) plane,B. sea star tilt angleβ versus time,C. Tube feet orientation,D. Tube feet length, andE. active forces generated along the tube feet versus time. The active force parameter is set toFmax = 1 for the results shown in the left column andFmax = 1.35 in the right column; all other parameters and initial conditions are kept the same; namely,mg = 1.5,γ = 50 and the feet are randomly oriented att = 0. The sea star exhibits a crawling motion forFmax =1 and it bounces forFmax =1.35. . . . . . . . . . . . . . 19 viii 3.4 Comparison between the crawl and bounce gaits. A. Frequency and amplitude of vertical oscillations, obtained by performing Fast Fourier Transform on the vertical positiony(t) of the sea star center of mass. In the bouncing case, vertical oscillations have a conspicuous frequency and large amplitude. B. Tube feet coordination order parameter versus time. Snapshot of the tube feet angles, mapped to the unit circle, are shown to the right att=50. In the bouncing case, the tube feet synchronize into two clusters, which results in a high value of the coordination order parameter. . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.5 Crawl gait: robustness to variations in tube feet initial coordination. We randomly perturb the initial conditions of the tube feet from the case shown in Fig. 3.3. The perturbations are chosen from a normal distribution with standard deviations increasing from0 to0.5 of the maximum angleθ max = π/ 3. For each standard deviation, we perform 20 simulations, each for a total integration time oft = 100. We report A. the percentage of the initial conditions that lead to unstable motion, and for the the initial conditions that lead to stable locomotion, we report B. the coordination order parameter, C. the total displacement in the x-direction, andD. the average vertical position. The black dots are the data points obtained from individual simulations, the line and the shaded area correspond to the average and standard deviation of the data points, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.6 Crawl gait: robustness to heterogeneity in the tube feet active forces. We randomly perturb the maximum active force Fmax in each tube foot for the crawling case shown in Fig. 3.3. Each tube foot is perturbed separately, allowing for a distribution of tube feet with heterogeneous force generation ability. For each foot,Fmax is chosen from a normal distribution with standard deviation equal to a fraction ofFmax = 1. We vary the standard deviation from 0 to 0.5 ofFmax = 1. For each standard deviation, we perform 20 simulations, each for a total integration time oft=100. We reportA. the percentage of the initial conditions that lead to unstable motion, and for the the initial conditions that lead to stable locomotion, we report B. the coordination order parameter, C. the total displacement in thex-direction, and D. the average vertical position. The black dots are the data points obtained from individual simulations, the line and the shaded area correspond to the average and standard deviation of the data points, respectively. . . . . . . . . . . . 21 3.7 Locomotion on wavy substrates. We place the sea star model of Fig. 3.3 on wavy substrates without changing the parameters of the model nor the control laws. The expression for the substrates is given byA. y =0.2sin(2πx ), andB.y =0.3sin(2πx/ 5). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.8 Locomotion on inclined stair-like substrates. We place the sea star model of Fig. 3.3 on inclined substrates without changing the parameters of the model nor the control laws. Two types of stairs are shown: A. stairs whose height and width are given by height= 0.5 and width= 5, leading to an average slope of5 ◦ , andB. height=0.2 and width=1, leading to an average slope of12 ◦ . . . . . . . . . . . . . . . . . . . . . . 22 3.9 Transition from crawling to bouncing: as a function of the maximum active force per tube foot, the sea star weight parameter spaces, and damping parameterγ showing (a)(mg,Fmax) forγ = 50, (b)(γ,F max) for mg =2, and (c)(γ,mg ) forFmax =1.5. In all cases the initial condition is the same as in Fig. 3.3. . . . . . 25 3.10 Cost of locomotion: for the parameter spaces shown in Fig. 3.9. The cost of locomotion is correlated with coordination order parameter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.1 Hierarchical structure for learning cooperative locomotion in tube feet. The RL policy is cloned among all the agents. Every agent observes local sensory cues and samples an action in response to this local observation. The same collective reward function is returned to all the agents. . . . . . . . . . . . . . . . . . . . . . 33 ix 4.2 Learning curves for learningA. the decision to detach andB. the decision to switch from active pushing to pulling. The bar plots show the ranking of the provided mechanosensory cues in terms of the average reward they gain during trainig. The orange curves in C. and D. show the control policy trained by RL using the most effective cue for the two learning tasks, respectively. The grey lines show our user-designed policies. E. Learning curve and F. the trained policy for learning both decisions of detachment and switching from pushing to pulling simultaneously. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.3 Testing performance of the control policies on 10 tube feet using A. the user-designed policy and B. the RL-trained policy. From left to right, the colormaps show displacement, success rate and cost of transport for body weight. The dashed lines mark the contour lines for the specified values. The insets show a smaller range of the parameter space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.4 Testing performance of the control policies on 100 tube feet using A. the user-designed policy and B. the RL-trained policy. From left to right, the colormaps show displacement, success rate and cost of transport for body weight. The dashed lines mark the contour lines for the specified values. . . . . . . . . . . . . . . . 37 4.5 Decentralized control of tube feet. A. Proposed architecture for a completely decentralized motor control of tube feet. The directionality command is added into the local control feedback loops. B. Symmetry-breaking from local control of tube feet. Histogram of the final horizontal position of a rigid body carried by 10 tube feet with a fixed recovery time τ =2 based on 1000 random simulations. In about80% of the cases, the tube feet can successfully carry the body to either left or right without any global directionality command. . . . . 38 6.2 Force magnitude versus dissipation time for a fixed plate in an oncoming flow. . . . . . . . 51 6.3 Force versus dissipation time for a flapping plate in an oncoming flow. . . . . . . . . . . . 51 7.4 Pairwise formation of interacting oscillators at different phase lags and lateral spacing. A. Equilibrium separation distances between the oscillators,B. power saving, andC. slope of the restoring fluid forces on a perturbed follower (right) as a function of phase lag and lateral spacing in a pair of interacting oscillators. For all the possible phase lags and all the shown lateral spacing, the pair reach equilibrium formations that are stable and power saving relative to a single swimmer. The contour lines show where the pairwise interactions lead to10%,20% and30% power saving. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 x Abstract Collective motion is widely observed in the animal kingdom from schooling fish to cooperative transport in ants. However, the underlying mechanism that give rise to collective motion are not always clear. Growing evidence suggest that the interaction of individual group members with the physical environment plays a key role in the emergence of collective behavior. Do group members actively seek collective motion? To what extent is it outcome of the passive physical interactions? In this thesis, we attempt to answer these questions in the context of two distinct model systems: (i) cooperative locomotion in tube feet inspired actuators and (ii) self-organization of flow-coupled swimmers. In the first part of this thesis, we propose a reduced-order model of the sea star body and tube feet biomechanics. We formulate hierarchical control laws that capture salient features of the decentralized nervous system in sea stars. Namely, in this model, there is no communication of sensory information between individual tube feet, but they are coupled mechanically through their structural connection to a rigid body. We find that these minimally coupled tube feet coordinate to generate robust forward loco- motion on various terrains and for heterogeneous tube feet initial conditions. We then use a model-free reinforcement learning approach to identify optimal mechanosensory cues for cooperative transport in this model. In the second part, we study the emergent swimming dynamics of flow-coupled flapping swimmers in the context of the vortex sheet model. We corroborate previous findings that mechanical coupling via the fluid medium only, with no sensory feedback, can create physical conditions that allow oscillating xi swimmers to self-organize into stable, energy saving formations. Importantly, we demonstrate that flow interactions lead to the emergence of dynamic and versatile formations of various spatial patterns, ranging from cooperative patterns that favor fair distribution of energy savings among group members to greedy patterns where few members get maximal benefit, leaving trailing swimmers with diminished opportuni- ties for maintaining group coherence and gaining energetic benefits. Our findings emphasize the role of the physical environment in the emergence of cooperative loco- motion in biology. They also offer a new paradigm for locomotion using minimally-coupled soft actuators and passive swimmers with potential applications to autonomous robotic systems. xii PartI Cooperativetransportusingtubefeetinspiredactuators 1 Chapter1 Introductionandmotivation Echinoderms are a group of marine invertebrates that use tube feet to achieve remarkable locomotion tasks. Sea stars, for example, have an oral surface that is lined with hundreds of tube feet used to crawl on various terrains, from smooth sand and glass surfaces to rocky substrates, see Fig.2.1. To achieve these feats of locomotion, individual tube feet are equipped with integrated sensing and actuation, and the activity of arrays of tube feet is orchestrated by a nervous system that is distributed throughout the body. How the distributed nervous system and numerous tube feet interact to give rise to coordinated motion has long been a question of interest for researchers. In 1945, Smith put forward a plan of neuron configuration and axon distribution based on behavioral experiments and neuroanatomy [117]. Lacking a brain, the central nervous system comprises a ring nerve at the center of the body with radial nerves that innervate the tube feet and extend to a simple eye at the distal tips of each arm and innervates the tube feet [15, 19, 85, 144]. The behavior of tube feet was studied later by recording the stepping phases – power and recovery strokes – that each tube foot undergoes during locomotion [64, 62, 63, 118]. While all tube feet step in the same direction during walking, Kerkut’s studies showed an absence of determinate phase relationship in the steps of different feet, suggesting the ability for individual action within each tube foot [98, 15]. Taken together, these experimental findings hint at the presence of a hierarchical structure within the nervous system of sea stars. There seems to be a central communication from the radial and ring nerves through 2 which a dominant direction of motion emerges, while the tube feet are individually capable of sensing and actuation. More recently, there has been a growing effort to understand distributed control in biology, in part due to their potential applications in autonomous robotic systems [83, 59, 96, 75, 140]. Specifically, there have been multiple studies on how a direction of motion emerges from the distributed nervous systems in echinoderms, such as brittle stars and sea urchins [5, 86, 141, 18, 144, 58]. These studies, although acknowledge the hypothesis of a hierarchical control mechanism in echinoderms, focus mostly on the centralized, system-level control, namely the directionality command and how it is transferred through the nerve ring. They lack details on how localized sensing and actuation at the tube feet level comes into play. More recently, there has been a growing effort to understand distributed control in biology, in part due to their potential applications in autonomous robotic systems [83, 59, 96, 75, 140]. Specifically, there have been multiple studies on how a direction of motion emerges from the distributed nervous systems in echinoderms, such as brittle stars and sea urchins [5, 86, 141, 18, 144, 58]. These studies, although acknowledge the hypothesis of a hierarchical control mechanism in echinoderms, focus mostly on the centralized, system-level control, namely the directionality command and how it is transferred through the nerve ring. They lack details on how localized sensing and actuation at the tube feet level comes into play. In Chapter 2 we discuss biomechanics of tube feet and propose a reduced-order force actuator model that captures salient features of the biomechanics. In Chapter 3, we introduce a mathematical model of sea star locomotion based on hierarchical control laws with local sensory-motor feedback loops at the tube foot level and a global directionality command at the system level. There is no explicit communication of mechanosensory information between tube feet. We then examine the sea star locomotion in the context of this mathematical model. We particularly focus on two distinct modes of locomotion exhibited by sea 3 stars: crawling and bouncing. When stimulated, sea stars across various species are reported to exhibit a bounce gait in which they coordinate their feet to increase their speed [53, 28, 26, 29]. This bounce gait is characterized by amplified vertical oscillations and a discernible frequency and wavelength of motion; see Fig. 2.1C. On the other hand, the crawl gait has a lower locomotion speed, dampened oscillations, and irregular trajectory of motion for which it is difficult to identify a frequency and wavelength. The bounce gait, which usually happens when tens of tube feet synchronize into two or three groups, raises new and interesting questions. Is there an underlying mechanism for sea stars to coordinate not only their direction of motion, but also the actuation of tens of tube feet? Or does the transition to bouncing happen as a result of the collective dynamics of individual and minimally-coupled tube feet? We address these questions by performing numerical experiments based on our mathematical model. Finally, in Chapter 4, we find optimal mechanosesory cues for local tube feet control using a deep Rein- forcement Learning (RL) approach. We find that shear and axial strains are the most effective mechanosen- sory cues for the tube feet to control the detachment and push/pull switch when in power stroke. Further, we introduce a modified control structure where the tube feet are controlled completely decentralized. Namely, the directionality, force generation and detachment are all part of a local sensory feedback loop in the tube feet. We show that by following these decentralized controllers, the tube feet will reach a con- sensus in a global direction of motion. We also show that our controllers are robust to the number of feet and can be generalized to more than a 100 tube feet. We first design physics-inspired control laws for the local sensory feedback control of the tube feet. Later, we optimize these local control laws and the mechanosensory cues used in them using a model- free deep Reinforcement Learning approach. Each tube foot is an autonomous entity that receives a global command about the direction of motion. Besides a shared directionality command, the tube feet are coupled only structurally through their attachment to a rigid body representation of the sea star. 4 Chapter2 Tubefeetbiomechanicsandmodel Many animals, including humans, have a rigid internal skeleton of bone that they use to support their bod- ies and generate forces. However, not all animals use jointed rigid jointed skeletons for force generation. Many use soft or flexible materials to support their bodies and move around. A common example of this is the softhydrostaticskeleton, found in biological structures such as human tongues and bodies of worms [27]. Echinoderms, such as sea stars, have specialized tube feet that use hydrostatic skeleton to generate forces for locomotion. In this chapter, we first introduce an overview of tube feet biomechanis. We then propose a reduced-order actuator model that captures the salient features of force generation in tube feet. 2.1 Tubefeetassofthydrostaticskeletons Tube feet move the sea star body by transmitting pressure-induced muscular forces through a soft body. This mechanism of force transmission is achieved by hydrostatic skeletons [27]. Hydrostatic skeletons (sometimes just called hydrostats) use a cavity filled with water; the water is incompressible, so the or- ganism can use it to apply force or change shape. The cavity is pressurized with muscle layers in the hydrostat’s walls. The most common muscle arrangement is to have a layer with lengthwise or longitudi- nal fibers and a layer with circular or circumferential fibers. Most hydrostatic skeletons are more or less 5 cylindrical, so longitudinal muscles will tend to shorten them and also widen them due to constant vol- ume, whereas circumferential muscles will tend to do the opposite. The longitudinal and circumferential muscles are thus antagonistic [1]. In sea stars, each tube foot consists of a cylindrical channel, called a podium, capped by a bladder-like structure, called an ampulla; see Fig. 2.1. The interior space of the ampulla is continuous with the interior of the podium, such that interstitial fluid moves freely between these two spaces. The walls of both the podium and ampulla includes layers of connective-tissue fibers that are stiff in tension (light blue lines in Fig. 2.1(E,F)), and superficial layers of muscle that serve to generate tension in the direction of the muscle fibers (orange lines in Fig. 2.1(E,F)). In the podium, the connective-tissue fibers are arranged helically to favor elongation of the podium under pressure, and the muscle fibers are arranged longitudinally [87]. The ampulla is characterized by longitudinally-oriented connective-tissue fibers and circumferential muscles. Experimental observations suggest that the podium is extended by contraction of the circumferential muscles in the ampulla. This action generates pressure that expels the interstitial fluid from the ampulla into the podium (Fig. 2.1f). Relaxation of the ampullar muscles causes the podium to retract. Retraction of the podium can be continued further through active contraction of the podium’s longitudinal muscles, which expels water from the podium into the ampulla. Further, a subset of these muscles could be ac- tivated to presumably bend the podium, provided the circumferential muscles of the ampulla maintain tension to prevent fluid flow from the podium. This model for the biomechanics of individual tube feet provides a starting-point for a mathematical description of these biological soft actuators and the premise for designing engineered counterparts. It is worth noting that the principles of operation of the tube feet as muscular hydrostats share simi- larities with pneumatic artificial muscles such as the McKibben actuators that convert hydraulic pressure into mechanical work. A mathematical relationship between the tensile forces and the length of these actuators can be obtained from first-principles [17, 123]. Similarly, force generation in the tube feet can 6 Figure 2.1: A. The common sea starAsteriasrubens (source: Shutterstock),B. close-up on the tube feet lining the ventral surface of Asterias rubens (source: Symbiotic service, San Diego), C. tube foot anatomy for an adult sea star, D. Activation of podia and ampulla muscles lead to contraction and extension of the tube feet. E. tube foot anatomy for an adult sea star, F. muscles are innervated by neurons located in the radial nerves and nerve ring. Activation of podia and ampulla muscles lead to contraction, extension and bending of the tube feet. be modeled by taking into account the balance between fluid pressure and wall stress in the ampulla and podium [88, 87]. 2.2 Mathematicalmodeloftube-feetinspiredactuators Consider a weight-carrying tube foot with the base of the podium attached to a flat horizontal plane. We assume that the tube foot cannot bend actively when attached; in other words, it cannot generate active moments during attachment, only active longitudinal forces. By contracting the ampulla and extending the podium, the tube foot produces an active pushing force; more precisely, by the law of action and reaction, the tube foot produces a pair of forces pushing onto both the plane of attachment and the load it is carrying. Inversely, an active pulling force can be generated by contracting the podium and expanding the ampulla. Clearly, active pulling requires additional contact forces to ensure the podium maintains contact with the ground, through friction, suction, or chemical adhesion [39, 92, 74]. This active force model can be thought 7 of as a state-dependent controller, where the magnitude and sign of the active force depends on the state of the tube foot, namely, its length and activation mode (pushing or pulling), while its direction is always acting longitudinally along the tube foot. In tandem with these active pushing and pulling forces, the tube foot experiences restoring elastic forces due to the connective tissues. Its extension or contraction is dampened by viscous resistance due to the interstitial fluid movement. Put together, each tube foot can be modeled as a soft actuator with (i) an active force generating elementF a that is either pushing or pulling, (ii) a passive restoring force elementF p , and (iii) a viscous damping elementF d , all acting along the length of the tube foot, as shown in the inset of Fig. 2.2A. Letl be the length of the tube foot, withl min andl max being its minimum and maximum length. We consider the restoring elastic force F p to be linear F p = − k p (l− l o ), where l o is the length at which the connective fibers are un-stretched. We also consider a linear damping force of the form F d = − c d ˙ l. Inspired by Hill’s muscle model [42, 32] (see Fig. 2.2), we use a piecewise linear force-length relation to model the active forceF a generated in the tube foot, namely, we write F a =F max Φ( l) (2.1) where F max is a scalar constant denoting the maximum force generated in the tube foot, and Φ( l) is a length-dependent function that describes the force profile. We let l c denote the length at which the active force is maximum as shown in Fig. 2.2. When in a pushing state,Φ( l) is given by Φ push (l)= (l− l min ) (l c − l min ) , l min <l <l c , (l− l max ) (l c − l max ) , l c <l <l max , 0, l <l min and l >l max . (2.2) 8 A B C 1 2 0 2 4 8 10 12 0 6 2 4 8 10 12 0 6 Active push/pull Active pushing Active pulling Fa Fp CE Fa Fp CE Hill’s muscle model Figure 2.2: Tube foot inspired soft actuator. A. when attached to a substrate, a tube foot generates either a pushing or a pulling force on the body it is attached to. The active force profiles (orange lines) are inspired by Hill’s muscle model (grey lines). Vertical motion of B. a point mass carried by a single vertical actuator and C. a rigid body of the same mass carried by an array of 10 identical actuators. Similar expressions can be obtained for pulling; the pushing and pulling force profiles are shown in Fig. 2.2 as a function of length. Here, the pushing and pulling force profiles are symmetric. Sea stars employ tube feet to generate a diverse array of motion. However, it is instructive first to explore the theoretical situation of vertical extension and contraction of a single tube foot carrying a weight mg, where m is mass and g is the gravitational constant. In this vertical “standing” regime, the length of the tube footl coincides with the vertical positiony of the mass; see Fig. 2.2(A). The equation of motion can be obtained from a straightforward application of Newton’s third law F a − k p (l− l o )− c d ˙ l− αmg =m ¨l. (2.3) Here, we introduced a parameterα = (1− ρ/ρ s ) to account for the buoyancy effects by considering the densitiesρ andρ s of water and the sea star, respectively, withρ/ρ s <1. The parameterα =(1− ρ/ρ s )∈ [0,1]: α = 1 corresponds to the dry weight of the sea star andα = 0 corresponds to a neutrally-buoyant sea star. Without loss of generality, we setα =1 while the value ofmg can be set independently. It is useful for writing the equations of motion in non-dimensional form to introduce the length scale L = l max − l min . We also introduce two time scales: an inertial time scale T g = p L/g obtained by 9 balancing the weight and inertial forces (mg∼ mL/T 2 g ) and a relaxation time scaleT d =c d /k p obtained by balancing the damping and passive spring forces (c d L/T d ∼ k p L). Small values ofT g describe a system where the weight is large compared to the inertial forces, whereas large values ofT d imply that damping is dominant. Observations of sea star locomotion suggest strong damping and weak inertial forces. We thus chooseT g <1 andT d >1 such that the non-dimensional ratioT d /T g is larger than 1. We rewrite Eq. (2.3) in non-dimensional form using the length scaleL=l max − l min , and the relaxation time scaleT d =c d /k p , µ ¨l+c d ˙ l+k p (l− l o )=F a − mg. (2.4) Here, all parameters and variables are non-dimensional. Specifically, c d = 1, k p = 1, and F a and mg are equal to the value of their dimensional counterparts divided by k p L. In (2.4), µ = mg/γ is a non- dimensional mass parameter, withγ =T 2 d /T 2 g =(c 2 d /k 2 p )/(L/g)≫ 1. We consider the active force elementF a generates either a contractile (pulling) or an extensile (push- ing) force as according to the following state-dependent control law: if the tube foot reaches a length l ≤ l min , the active force is zero and the tube foot cannot contract further, the controller requires that it extends by producing a pushing forceF a following the profile in Fig. 2.2 shown in solid line. Alternatively, ifl≥ l max , the controller requires the tube foot to contract by producing a pulling forceF a following the profile in Fig. 2.2 shown in dashed line. We rewrite Eq. (2.4) in light of this state-dependent controller: the expression for F a switches from pushing to pulling and vice-versa, depending on the state of the tube foot. We employ a change of variable froml toℓ defined as follows ℓ= l− l min , pushing, l max − l, pulling. (2.5) 10 The expressions forΦ push andΦ pull , when expressed in terms ofℓ satisfy the symmetry property:Φ push (ℓ)= − Φ pull (ℓ)=Φ( ℓ), which follows directly from (2.2) and (2.5), Φ( ℓ)= ℓ L− ∆ , 0<ℓ<L− ∆ , L− ℓ ∆ , L− ∆ <ℓ<L, 0, ℓ<0 and ℓ>L. (2.6) Here, ∆ denotes the change in length from where the active force is maximum to where it decays to zero; see Fig. 2.2. Namely, ∆ = l max − l c when pushing and ∆ = l c − l min when pulling, and by the symmetry property considered here, both values are equal. We also introduceδ = l o − l min for pushing andδ =l max − l o for pulling, which we take to be equal. We get a simplified expression of Eq. (2.4) during pushing and pulling, µ ¨ℓ+c d ˙ ℓ+k p ℓ=F max Φ( ℓ)+k p δ ∓ mg, (2.7) Here,− mg is for pushing and+mg is for pulling. Eq. (2.7) and Fig. 2.2(B, C) have several important consequences. The most important is that the weight acting on the tube foot breaks the extensile/contractile symmetry of the actuator: when standing on a horizontal flat surface, gravity aids the tube foot during contraction and acts against it during extension. Active pushing forces are imperative to carry the sea star weight but tube feet can be made to contract passively under the gravity. Indeed, experimental observations suggest that sea stars relax from actively pulling by allowing their tube feet to buckle passively under weight. When pushing and pulling are both active as in the model considered here, a weight-carrying tube foot takes a longer time to fully extend from l min tol max than to fully contract froml max tol min . This is shown in Fig. 2.2(B,C) for a vertical tube foot carrying a point mass and ten vertical tube feet attached to a rigid body of the same weight. Lastly, the 11 vertical oscillations afforded by Eq. (2.7) are unstable to all non-vertical perturbations unless multiple tube feet are put to work together as shown later. 12 Chapter3 Tubefeetinspiredlocomotion We study the dynamics of a rigid body carried by the tube feet inspired actuators. To this end, we first introduce our model of the sea star body mechanics and how it interacts with the tube feet. We then propose a hierarchical control model for the actuation of the feet that is inspired by biological observations and the physical properties of the model. Finally, we show locomotion results of a rigid body carried by ten tube feet inspired actuators. 3.1 Seastarsbodymechanics We model the sea star as a rigid body of mass m connected to a series of N tube feet separated by a constant distanced, as shown in Fig. 3.1A. Let(x,y) denote the position of the center of mass of the sea star in inertial frame (e x ,e y ), and β denote its tilting angle measured from the x-axis in the counter- clockwise direction. The signed position of the base point of each tube footn relative to the sea star center of mass isd n , such thatd n+1 − d n = d,n = 1,...,N. The kinematic state of each tube foot is described by its lengthl n and inclination angleθ n measured from they-axis in the counter-clockwise direction. 13 The balance laws for the forces and moments acting on the sea star body are given by x-dir: − c x ˙ x+ X n F n sinθ n =µ ¨x, y-dir: − c y ˙ y− mg+ X n F n cosθ n =µ ¨y, tilt: − c β ˙ β + X n F n d n cos(θ n − β )=I ¨ β. (3.1) whereI is the moment of inertia of the sea star body andc x ,c y , andc β are the internal translational and rotational damping parameters, all expressed in dimensionless form. Here, to simplify the problem, we don’t compute the damping force c d ˙ l n exerted by individual tube feet. Instead we account for external damping effects from the environment in terms of lumped damping parameters c x ,c y andc β . The forceF n exerted by tube footn on the sea star body acts along the direction of the tube foot, F n =F a,n − k p (l n − l o ). (3.2) The active forceF a,n of tube footn is either a pushing or pulling force depending on its statel n andθ n . The active force profile follows directly from Eqns. (2.1,2.2) and it is depicted in Fig. 2.2A. To close the system of equations (3.1) and (3.2), note that the tube feet exert forces on the sea star body only when they are attached to the ground, that is to say, during the tube foot power stroke. When attached, the state(l n ,θ n ) of the tube feet must satisfy the following constraint equations x n − l n sinθ n =x+d n cosβ, l n cosθ n =y+d n sinβ, (3.3) wherex n denotes the location of attachment of tube feetn on the ground. In this formulation, the length and orientation of the tube feet during attachment are slaved to the position and orientation of the sea star 14 Table 3.1: Sea star parameters (based on [116, 64, 2]) adult Asterias rubens body diameter 10–30 cm wet weight 3.25–6 g dry weight 9–15 g Number of tube feet ≈ 1000 Tube feet length 1.25–8 mm force per tube foot weight/0.1× (number of tube feet) body. Eqns. (3.1), (3.2) and (3.3) form a differential-algebraic system of 3+2N equations for3+2N un- knowns(x,y,β,l n ,θ n ) provided that we define control rules for the tube feet attachment and detachment as discussed in Section 3.2. 3.2 Hierarchicalstructurefortubefeetcontrol We propose a hierarchical motor control of the tube feet consisting of global and local components: (i) a global directionality command – descending from the nerve ring and radial nerve – responsible for communicating the step direction to all tube feet [117], and (ii) local sensory-motor feedback loops at the individual tube feet level that dictate the power and recovery stroke of the tube foot, that is to say, the decisions to push or pull and attach or detach. The only coupling between tube feet is via their structural attachment to the sea star body, as depicted schematically in Fig. 3.1B. We implement the control law with the aforementioned global-local characteristics into Eqns. (3.1), (3.2) and (3.3) as follows. At the global sea star level, all actuators are directed using an open-loop control command that specifies the step direction e; here the step direction is either in the negative or positive x- directione=± e x . At the local tube feet level, each actuator senses its own state(l n ,θ n ) and accordingly decides to push, pull, or detach and reattach. The local state-dependent control law can be summarized 15 (x, y) β θ l d Sea star body Local sensory-motor feedback loops Mechanical coupling Detach Attach Pull/Push Detach Attach Pull/Push Detach Attach Pull/Push Global directionality command Extension Contraction Fa Fp Fa Fp Pushing Pulling Tube feet Control laws Mechanical model Tube foot model CE CE x y A B Figure 3.1: Mathematical model of locomotion. A. schematic of our mechanical model of the sea star and tube feet inspired actuators, with inset showing contractile, passive and dissipative force elements along each tube foot, B. hierarchical motor control of the tube feet consisting ofglobal directionality commands issued by the radial nerves and nerve ring andlocal sensory- motor feedback loops at the tube foot level. as follows. In the power stroke phase, for l n < l max , the actuatorn decides to push or pull based on its orientationθ n relative to the direction of motion. Forl n <l max : sinθ n e x · e>0: pull, sinθ n e x · e<0: push. (3.4) Whenl n >l max , the actuator detaches, takes a step of size∆ θ n in the direction of motion, then reattaches to the ground. These actions constitute the recovery stroke phase. The duration of the recovery stroke, the period from detachment to reattachment, is denotedτ n . Forτ n =0, the reattachment satisfies x + n =d n +l + n sin∆ θ n e x · e, l + n = l − n cosθ n cos∆ θ n . (3.5) Here,l − n andl + n denote the length of the tube foot right before and right after its recovery stroke, andx + n is the point of attachment of the base of the tube foot right after recovery. 3.3 Pointmasscarriedbytwotubefeet To illustrate the hierarchical, state-dependent controller, we apply it first to the simple example of a point mass connected to two tube feet joined at their based = 0, as shown in Fig. 3.2A. The two tube feet are 16 initially oriented such that one tube foot is in a pushing state and the other in a pulling state. We set F max = 2, mg = 1, γ = 10, and c x = c y = 1. The step size ∆ θ = π/ 6 is equal for both feet, and the feet have characteristic lengths l min = 1, l max = 2, l o = 1.5 and l c = 1.9. We follow the hierarchical control laws detailed in §3.2: both feet are instructed to step in the positive x-directione= e x . Other than this global directionality command, all details of the power stroke and the transition to recovery stroke (all decisions to push or pull, or to detach and reattach) are done locally, at the tube foot level. There is no communication between the two feet other than their mechanical coupling via their attachment to the same mass. We solve the differential-algrebraic system of equations (3.1) and (3.3) numerically, where the active component ofF n in Eq. (3.2) is dictated by the state(l n ,θ n ) of each tube foot (n = 1,2). Although the controller does not explicitly impose a coordination pattern between the two feet, a clear anti-phase coordination emerged in time, and the body oscillated in the vertical direction and moved forward in the horizontal direction. The anti-phase coordination is reflected in the angles of the tube feet and the active forces shown in Fig. 3.2(B,C). This walking motion is fundamentally distinct from existing models of bipedal walking [22, 21, 122]: (i) the feet here are “soft” in the sense that they offer no resistance to bending, nor do they produce active moments during attachment; they only produce and sustain longitudinal forces along the foot length; (ii) there is no prescribed time period for attachment; the duration of each attachment cycle emerges from the state-dependent controller; (iii) the controller itself imposes no a priori coordination between the feet. Each tube foot follows its own local sensory-motor control feedback loops, without information about the state of the other foot; coordination emerges from mechanical coupling to the point mass. We next expand on these ideas in the context of arrays of soft actuators. 3.4 Locomotionusinganarrayoftentubefeet We investigate the motion of the sea star model connected to ten tube feet. Specifically, we model the sea star as a rigid body, with massµ and moment of inertiaI = 0.04µD 2 , whose shape is reconstructed 17 -2 2 0 B C Fa θ ̟/3 -̟/3 -̟/6 ̟/6 0 y A 3 2 1 0 x 0 5 10 15 20 t = 0 t = 15 t = 30 0 10 20 30 time, t 0 10 20 30 time, t Figure 3.2: Bipedal locomotion. A point mass attached to two tube feet, each producing longitudinal pushing or pulling forces along the direction of the foot in the attached phase (power stroke) and taking a step forward in the detached phase (recovery stroke). A. Trajectory of the point mass in the(x,y) plane and snapshots of the walker at three instants in time. B. Orientation angles of the tube feet versus time. C. Active forces along the tube feet as a function of time. Positive force corresponds to pushing and negative force corresponds to pulling. The parameter values are set toFmax = 2,mg = 1 andγ = 10. The step size taken after a detachment-reattachment cycle is∆ θ =π/ 6. from a side view image of an actual sea star. The sea star damping parameters are set to c x = c y = 1, c β = 10. The tube feet are aligned in a single line, separated by distance d = 1, as shown in Fig. 3.1A. The length parameters and step size of the tube feet are held at the same values as above throughout this chapter. We explore the behavior of the sea star model as a function of the maximum active forceF max per tube foot, the sea star weightmg, and the intrinsic damping parameterγ . We emphasize that the tube feet are modeled as massless actuators, that sustain and produce longitudinal forces only, with no additional constraints to prohibit intersection between neighboring feet. The behavior of the sea star body and tube feet is shown in Fig. 3.3, formg = 1.5, γ = 50, F max = 1 (left column) and F max = 1.35 (right column), both starting from zero initial velocity and the same randomly-oriented feet. WhenF max = 1 (left column), the sea star moves in thex-direction, with small vertical and angular oscillations reminiscent of the crawl gait observed in actual sea stars. ForF max =1.35 (right column) the mode of locomotion is reminiscent of the bounce gait observed in sea stars and shown in Fig. 2.1C [53, 28, 26, 29]; namely, it is characterized by a distinguishable bounce frequency at the sea star level and two anti-phase clusters of tube feet, resembling the bipedal locomotion in Fig. 3.2. A Fast Fourier Transform of the dominant frequencies and amplitudes of vertical oscillations clearly indicate the increase in amplitude and the existence of a dominant frequency of oscillations in the bounce gait, see Fig. 3.4A. 18 0 10 20 30 40 50 0 10 20 5 15 time, t Fa 1.3 -1.3 0 bouncing sinθ 0 0.5 -1 -0.5 length 2 1 1.5 x 2 1.5 1 0.5 0 10 15 35 y β 0 0 10 20 5 15 time, t foot index 1 5 10 -π/18 π/18 foot index 1 5 10 1 5 10 30 25 20 5 x A B C D E foot index crawling Figure 3.3: Crawl and bounce gaits: of a sea star model with ten tube feet. A. Trajectory of the sea star center of mass in the (x,y) plane,B. sea star tilt angleβ versus time,C. Tube feet orientation,D. Tube feet length, andE. active forces generated along the tube feet versus time. The active force parameter is set toFmax =1 for the results shown in the left column andFmax =1.35 in the right column; all other parameters and initial conditions are kept the same; namely,mg = 1.5,γ = 50 and the feet are randomly oriented att=0. The sea star exhibits a crawling motion forFmax =1 and it bounces forFmax =1.35. In crawling and bouncing, the tube feet start from the same initial orientation with no clear coordina- tion between them in the first few steps. But, as time progresses, a coordination pattern emerges solely from the mechanical coupling between the tube feet and the sea star body. The coordination pattern is not restricted to adjacent feet, and it differs substantially between the crawling and bouncing gaits, as clearly reflected in the plots of sinθ n , lengthl n , and active forceF a,n along each tube foot (n=1,...,10) shown in Fig 3.3(C-E). The tube feet are labeled consecutively such that two feet with labelsn andn+1 are adjacent. The feet develop a coordination pattern in time that is not restricted to adjacent feet; in the crawling motion, tube feet 2, 7 and 10 coordinate their motion while in the bouncing motion, tube feet 2, 3, 6, 7 and 9 coordinate their motion. The active forces generated in the crawling gait are weaker. The duration of the power stroke (time from attachment to detachment) is approximately 35% longer in the 19 0 10 20 30 40 50 time, t 0 0.2 1 0.8 0.6 0.4 bouncing coordination order parameter 0 1 2 3 4 5 0.1 0.2 0 0.15 0.05 bouncing crawling Amplitude Frequency A bouncing crawling crawling B Figure 3.4: Comparison between the crawl and bounce gaits. A. Frequency and amplitude of vertical oscillations, obtained by performing Fast Fourier Transform on the vertical positiony(t) of the sea star center of mass. In the bouncing case, vertical oscillations have a conspicuous frequency and large amplitude. B. Tube feet coordination order parameter versus time. Snapshot of the tube feet angles, mapped to the unit circle, are shown to the right att=50. In the bouncing case, the tube feet synchronize into two clusters, which results in a high value of the coordination order parameter. crawling gait than in the bouncing gait, which is consistent with our experimental observations (results not yet published). To quantify the degree of coordination and highlight the difference in coordination between crawling and bouncing, we sort the tube feet into subsets, or clusters, that contain tube feet of similar inclination anglesθ n ; namely, tube feet of anglesθ n within an angular toleranceϵ =π/ 50 from each other belong to the same cluster. The number of clustersN c lies in the range2≤ N c ≤ N. The caseN c =1 is equivalent to a single tube foot, which cannot stably carry a weight and move forward. ForN c =2, the tube feet are coordinated into two groups. ForN c =N, the feet exhibit maximum disorder. The degree of coordination is measured via a coordination order parameter defined as p(t) = 2/N c (t), wherep(t) ∈ [0.2,1]; p = 1 corresponds to the tube feet split in two clusters, exhibiting the highest degree of coordination for stable locomotion (similar to bipedal locomotion), whereas lower values ofp indicate larger number of clusters and lower degree of coordination. In Fig. 3.4B, we plot the (time-averaged) coordination order parameter p(t) as a function of time for the two examples in Fig. 3.3. In the bouncing gait, the coordination order parameter converges to 1 while in the crawling gait it hovers around approximately0.3. By way of visualization, we map the inclination angle of each tube foot to a point on the unit circle,z n (t)=e iθ n(t) , forn=1,...,N, wherez n (t) indicates 20 1 0 .8 .6 .4 .2 failure rate % 100 0 25 50 75 250 200 150 100 50 0 standard deviation, i.c. 0 0.5 0.4 0.1 0.2 0.3 standard deviation, i.c. 0 0.5 0.4 0.1 0.2 0.3 standard deviation, i.c. 0 0.5 0.4 0.1 0.2 0.3 standard deviation, i.c. 0 0.5 0.4 0.1 0.2 0.3 x-displacement order parameter y-displacement 1.5 0 1 .5 coordination A D C B Figure 3.5: Crawl gait: robustness to variations in tube feet initial coordination. We randomly perturb the initial conditions of the tube feet from the case shown in Fig. 3.3. The perturbations are chosen from a normal distribution with standard deviations increasing from0 to0.5 of the maximum angleθ max =π/ 3. For each standard deviation, we perform 20 simulations, each for a total integration time oft = 100. We reportA. the percentage of the initial conditions that lead to unstable motion, and for the the initial conditions that lead to stable locomotion, we reportB. the coordination order parameter,C. the total displacement in thex-direction, andD. the average vertical position. The black dots are the data points obtained from individual simulations, the line and the shaded area correspond to the average and standard deviation of the data points, respectively. failure rate % 250 200 150 100 50 0 standard deviation, Fa 0 0.5 0.4 0.1 0.2 0.3 standard deviation, Fa standard deviation, Fa 0 0.5 0.4 0.1 0.2 0.3 standard deviation, Fa x-displacement order parameter y-displacement 1.5 0 1 .5 A D C B coordination 100 0 25 50 75 0 0.5 0.4 0.1 0.2 0.3 0 0.5 0.4 0.1 0.2 0.3 1 0 .8 .6 .4 .2 Figure 3.6: Crawl gait: robustness to heterogeneity in the tube feet active forces. We randomly perturb the maximum active force Fmax in each tube foot for the crawling case shown in Fig. 3.3. Each tube foot is perturbed separately, allowing for a distribution of tube feet with heterogeneous force generation ability. For each foot,Fmax is chosen from a normal distribution with standard deviation equal to a fraction ofFmax = 1. We vary the standard deviation from0 to0.5 ofFmax = 1. For each standard deviation, we perform 20 simulations, each for a total integration time oft = 100. We reportA. the percentage of the initial conditions that lead to unstable motion, and for the the initial conditions that lead to stable locomotion, we reportB. the coordination order parameter, C. the total displacement in thex-direction, and D. the average vertical position. The black dots are the data points obtained from individual simulations, the line and the shaded area correspond to the average and standard deviation of the data points, respectively. the position of thenth actuator in the complex plane. Note that the range of angles of the tube feet covers a small portion of the unit circle, since we fixed the step size to π/ 6. To make the clusters more discernible, we rescale θ n to πθ n (t)/θ max to lie in the range [0,2π ]. Here, θ max is the maximum inclination angle reached in a given simulation. A depiction of the scaled tube feet angles on the unit circle is shown for a snapshot att = 50 in Fig.3.4(b); clearly in the bounce gait, the tube feet angles belong to two clusters, where the feet in the same cluster are not necessarily adjacent spatially. 21 0 10 20 30 40 x 4 2 -2 0 y 0 10 20 30 40 x 4 2 -2 0 y t = 20 t = 0 t = 20 t = 0 (a) (b) Figure 3.7: Locomotion on wavy substrates. We place the sea star model of Fig. 3.3 on wavy substrates without changing the parameters of the model nor the control laws. The expression for the substrates is given by A. y = 0.2sin(2πx ), and B. y =0.3sin(2πx/ 5). t = 0 t = 20 0 10 20 30 40 0 5 10 y x (a) (b) 0 10 20 30 40 x 0 5 10 y t = 0 t = 20 Figure 3.8: Locomotion on inclined stair-like substrates. We place the sea star model of Fig. 3.3 on inclined substrates without changing the parameters of the model nor the control laws. Two types of stairs are shown: A. stairs whose height and width are given by height= 0.5 and width= 5, leading to an average slope of5 ◦ , and B. height= 0.2 and width= 1, leading to an average slope of12 ◦ . 22 Robustness of locomotion We gauge the robustness of the crawling behavior shown in Fig. 3.3(left column) to variations in the parameters of the tube feet. To this end, we perturb the initial conditions of the tube feet randomly from a normal distribution with mean values centered at the initial conditions in Fig. 3.3. We vary the standard deviation from0 to50% of the maximum possible initial inclination angle θ max = π/ 3. This value ofθ max is set such that it automatically ensures thatl n (0)≤ l max , for alln. For each standard deviation, we perform Monte Carlo simulations with 20 random initial conditions. For a fraction of initial conditions, the sea star fails to produce stable forward movement. We report the failure rate in Fig.3.5A. The failure rate tends to increase as the standard deviation of the noise increases. For the initial conditions that produce stable locomotion, we quantify the total horizontal displacement of the body at end of the integration time, as well as the average vertical position and average coordination order parameter, both averaged over the period fromt = 80 tot = 100. The results are shown in Fig.3.5(B-D), where the black dots represent individual realizations of the Monte Carlo simulations, while the solid lines and shaded areas correspond to the mean and standard deviation of the results. It is clear from the tight standard deviations in thex- andy-displacements that the overall locomotion of the sea star is robust to variations in initial conditions, even when the details of the tube feet coordination varies. We next explore the robustness of locomotion to heterogeneity in the tube feet actuation. Namely, we vary the active force in each tube foot independently, by choosingF max for each tube foot randomly from a normal distribution with mean value centered atF max =1 and a standard deviation ranging from 0 to50% ofF max ; that is, the magnitude of the active forces produced in each tube foot varies across all ten tube feet. The results of these variations on the overall sea star behavior and tube feet coordination are shown in Fig.3.6. Similar to variations in initial condition, the failure rate generally increases with increasing standard deviation. However, in comparison to variations in initial conditions, heterogeneity in F a across tube feet produces larger variations in the tube feet coordination as well as in the overall displacement of the sea star body. 23 We next comment on the robustness of the crawling motion to variations in the substrate itself. We consider the sea star with the same parameters values and initial conditions shown in Fig. 3.3 (left panel), and we investigate its ability to crawl on wavy terrains in Fig. 3.7 and up stair-like terrains in Fig. 3.8. The wavy substrate is described by a sinusoidal function of amplitude a = 0.2 and wavelength λ = 1 in Fig. 3.7A anda = 0.3, λ = 5 in Fig. 3.7B. The stairlike terrain is described by stair width w = 5 and height h=0.5 in Fig. 3.8A and w=1, h=0.25 in Fig. 3.8B. In all cases, the sea star moves robustly with adjustments made neither to the control model itself, nor to the mechanical parameters. This robustness is mediated by the decentralized local sensory-motor feedback loops at the individual tube foot level, where the control action itself depends on the state of the tube foot. A few comments on the robustness of the bouncing gait are in order. By conducting similar numerical experiments (see supplemental movies S6-9), we found that the bouncing gait is robust for weak noise (standard deviation≤ 10%− 15%) and weak perturbations in the substrate. For larger values of noise or substrate perturbations, the distinct bouncing frequency is lost and the trajectories of stable locomotion resemble the crawling gait, albeit at the higher value ofF max =1.35. Exploring the parameter space Last, we analyze the locomotion modes on flat horizontal terrains as a function of the maximum active force F max per tube foot, the sea star weight mg, and the sea star damping parameter γ . Specifically, we look at three cross-sections of the three-dimensional parameter space(F max ,mg,γ ), while keeping the initial conditions and all other parameter values as in Fig. 3.3. In Fig. 3.9A, we investigate the sea star behavior as a function of F max and mg. For weak tube feet (tube feet whereF max is small), the motion is unstable and the sea star can neither crawl nor bounce. AsF max increases for a given mg, the sea star first crawls, then transitions to a bouncing mode, provided that the weight exceeds a minimum value. This suggests that inertial effects, though small, seem necessary for the bouncing motion to appear. The transition from crawling to bouncing happens abruptly with the coordination order parameter increasing sharply to 1. As F max increases further, the motion becomes 24 2 0.5 1.5 1 0.5 3 2 0.5 1.5 1 1.5 2 1 2.5 100 60 80 40 20 3 2 0.5 1 1.5 2.5 0 100 60 80 40 20 0 sea star weight, mg active force, Fmax active force, Fmax sea star weight, mg damping parameter, γ damping parameter, γ crawling unstable bouncing (a) (b) (c) Figure 3.9: Transition from crawling to bouncing: as a function of the maximum active force per tube foot, the sea star weight parameter spaces, and damping parameterγ showing (a)(mg,Fmax) forγ = 50, (b)(γ,F max) formg = 2, and (c)(γ,mg ) for Fmax =1.5. In all cases the initial condition is the same as in Fig. 3.3. unstable again, implying that, for stable locomotion, the maximum active force per tube foot should be bounded between an upper and a lower limit. The lower limit seems to increase linearly withmg for light sea stars and becomes independent of the weight as the sea star weight exceedsmg ≈ 1.75. Meanwhile the upper limit seems to increase linearly withmg, with an approximate slope of0.7. The sea star behavior as a function of F max and γ , for mg = 2, exhibits similar trend in the transi- tion from crawling to bouncing asF max increases, see Fig. 3.9B. Once again, we observe that in order to achieve stable locomotion, F max should be bounded above and below. The importance of inertial effects for bouncing is clear in these results as well. Asγ increases, inertial effects decrease, inducing a transition back to crawling for a given value ofF max . The sea star behavior as a function ofmg andγ , forF max = 1.5, is shown in Fig. 3.9C. The behavior is consistent with the previous observations: increasingγ decreases the inertial effects and decreases the region of the parameter space where bouncing occurs. Further, for a given γ , at lower load mg, the sea star bounces but as mg increases, it transitions to crawling, similar to the effect of increasing mg for a constantF max in Fig. 3.9A. 25 2 0.5 1.5 1 0.5 3 2 0.5 1.5 1 1.5 2 1 2.5 100 60 80 40 20 3 2 0.5 1 1.5 2.5 0 100 60 80 40 20 0 sea star weight, mg active force, Fmax active force, Fmax sea star weight, mg damping parameter, γ damping parameter, γ (a) (b) (c) cost of locomotion 5 2 4 3 ×10 -2 Figure 3.10: Cost of locomotion: for the parameter spaces shown in Fig. 3.9. The cost of locomotion is correlated with coordi- nation order parameter. To examine the energetic cost of the bouncing and crawling gaits, we define cost of locomotion as the (time-averaged) active power input by all tube feet per horizontal distance traveled by the sea star, namely, cost of locomotion= ⟨P a ⟩ x-distance , (3.6) whereP a = P n F a,n ˙ l n . We compute the cost of locomotion for the results in Fig. 3.9, shown separately in Fig. 3.10 for clarity. The bouncing gait is correlated with a higher cost of locomotion, implying a trade-off between speed and efficiency. Bouncing gaits are characterized by higher speeds and also higher costs, which implies lower efficiency. 3.5 Discussion This chapter of our study examined the control laws that underly locomotion in sea stars, as a model system for the control of distributed sensors and actuators. Sea stars use hundreds of tube feet to walk over various terrains. The tube feet seem to coordinate the direction of their power stroke, regardless of their arm’s position, with the direction of walking, whereas the power and recovery strokes of individual tube feet seem to be governed locally at the tube foot level. Here, we developed a mathematical model of each tube foot as a soft actuator, consisting of active, passive, and dissipative force elements, that can 26 actively extend or contract, generating active pulling or pushing forces on the substrate and the sea star body. We then studied the dynamics of the sea star driven by an array of such soft actuators. The tube feet were actuated according to a hierarchical motor control, where the direction of motion is globally communicated to all tube feet, while each foot is actuated according to local sensory-motor feedback loops. In these feedback loops, the feet use minimal sensory information (their own inclination angle and length) and generate active forces accordingly. The feet are coupled only mechanically through their structural connections to the sea star body. We found that the collective effect of the tube feet can lead to stable crawling motion of the sea star body. The model also exhibited robustness to perturbations in initial condition and heterogeneity in the ability of the tube feet to generate active forces, as well as to irregularities in the substrate geometry. Recent reports show that as a part of their escape response, sea stars can coordinate their numerous tube feet, in a gait known as bouncing, to increase their speed of locomotion [53, 28, 26, 29]. We hypothe- sized that this transition to bouncing can occur in the context of the same hierarchical motor control used in crawling. To test this hypothesis, we varied the maximum active forceF max per tube foot, the sea star weightmg, and the sea star damping parameterγ . We identified a major transition in the coordination of the tube feet as we increasedF max and decreasedmg andγ . These transitions are invariably associated with an increase in the active work done by the tube feet relative to the work dissipated due to damping or required to lift the weight of the sea star. During bouncing, the tube feet synchronized into two clusters, which is clearly reflected in the temporal evolution of their inclination angles, lengths, and active force. The clusters are not restricted to adjacent tube feet. Moreover, the vertical oscillations of the body were amplified, and followed a discernible frequency and wavelength; which are characteristics observed in the bounce gait in sea stars. We quantified the level of coordination in the tube feet, by introducing a coor- dination order parameter that takes values between 0.2 and 1. The coordination order parameter varied between 0.2 and 0.5 in the crawling motion, and stayed near1 in the bouncing motion. 27 To understand why the bounce gait is a part of the sea stars escape response as opposed to their normal mode of locomotion, we computed the cost of locomotion of the crawl and bounce gaits. We defined the cost of locomotion as the average active power consumed per horizontal distance traveled during a specific locomotion time. We found a strong correlation between the coordination order parameter and the cost of locomotion. More specifically, we found that higher tube feet coordination, characteristic of the bounce gait, consumes more power and therefore comes at a higher cost. This suggests that although the bouncing motion can increase the speed of locomotion in sea stars, it is not always favorable for them in terms of power consumption. A few comments on the advantages and limitations of the mathematical model are in order. Our low order model intimately couples the neural sensory-motor control to the physical system and its action on the environment, i.e, substrate. This approach is consistent with the theme of “embodied intelligence” or “embodiment” [107, 50, 49, 76]. It reflects essential elements in the current understanding of how sea stars control locomotion based on neuroanatomy and behavior experiments [117, 64, 62, 63] in the form of a higher level representations of the neural circuits underlying locomotion as feedback control laws. How- ever, our model does not describe the details of the physiology, connectivity, and activity of these neural circuits [13, 8]. From a mechanical standpoint, our model neglects many of the complications in sea stars, including details of the tube feet biomechanics as muscular hydrostats [66, 65, 88] and deformations along the arms [10, 33, 102, 71, 16]. Another limitation of this study is that it considers a two-dimensional model to study locomotion in one dimension. Future extensions of this work will include the more complicated dynamics required to undertake turning maneuvers. We close by noting that gait transitions, reminiscent to the transition from crawling to bouncing re- ported here, are observed in various forms of animal locomotion including the walking to running transi- tion in humans. In insects, a transition from tetrapod to tripod motion is observed when walking at higher stepping frequencies. In the tripod gait, the legs coordinate into two groups: three legs in contact with 28 the substrate and three in a swing phase [25, 7]. Centipedes also use numerous feet to locomote [140], and although the underlying mechanisms for force generation are fundamentally distinct from those of sea star tube feet, the two systems exhibit similarities in the spatiotemporal patterns of attachment and detachment that are worth exploring in future works. 29 Chapter4 Learningoptimalmechanosensorycuesforcooperativelocomotion We showed in Chapter 3 that the proposed designed hierarchical control structure performs well under desperate conditions for cases of two and ten tube feet. However, the optimally of the proposed control laws and the machnosensory cues were not probed. Furthermore, we only tested the policy on two and ten tube feet, however adult sea stars are usually equipped with hundreds of tube feet. Here, we use a deep Reinforcement Learning (RL) algorithm to train optimal laws for tube feet control, while also investigating the effectiveness of different mechanosensory cues in training. It should be noted that this chapter is an ongoing study and will be updated with more results in the near future. 4.1 TubefeetcontrolviaReinforcementLearning We use a model-free RL algorithm to identify tube feet control laws that are optimal for stable forward locomotion. Specifically, assuming a global directionality command, we divide the local mechanosensory feedback loops for each tube foot into 2 decisions: (i) when to detach and transition into the recovery stroke, and (ii) when to switch from active pushing to active pulling while in the power stroke. We begin by training the tube feet on each one of these control decisions separately, while prescribing the other decision according to our designed control laws in Section 3.2. This approach allows us to discover optimal mechanosesnroy cues for each decision task in the feedback loop at relatively low computational cost. 30 Finally, the tube feet are trained on both decision simultaneously using these optimal mechanosesnry cues and develop a complete mechanosesnsory feedback controller. In our implementation of the RL framework, the model is divided into the tube feet controlled bylearn- ing agents and the environment which consists of rest of the physical world (sea star body and substrate). Next we introduce these notions that are central to any RL implementation: state of the system, theobser- vations given to the learning agent, the actions taken by the agent, and the rewards given to the agent in light of its behavior. State The state s t of the system in our model at time t is fully described by the body’s position and orientation in inertial frame (x,y,β ) and the tube feet distribution on the body, their lengths and tilt angles(l n ,θ n ); see Fig. 3.1A. Observation The observations here, refer to the mechanosensory cues of the tube feet. In our model the observations are egocentric, meaning that every agent (tube foot) only senses information about its own state (length, angle, forces, ...) and does not communicate in anyway with the neighboring agents (except mechanically). The observations are discussed in detail in Section 4.2. Action The agents have different actions for each control decision. For decision (i), the action for each tube foot is whether to detach from the substrate and enter the recovery stroke or remain attached. For decision (ii), every foot has control over whether to generate pushing or pulling forces while in attachment to the substrate. Note that in both cases the actions are binary decisions. Rewardandpolicy In RL, the decision making process is modeled as a stochastic control policyπ θ (a t |o t ) that produces actionsa t given observationso t of the tube feet. The policy is parameterized by a set of pa- rametersθ to be optimized. An optimal policy is learned to produce behavior that maximizes rewards. We use a dense shaping reward, that is, the agents are given a reward at every decision time step. Specifically, 31 we use a reward function that encourages the tube feet to carry the sea star body forward while avoiding crashing into or moving too high off the substrate. After several trials, we found the most effective reward function to ber t =|x t |˙ x t (− exp(y t − y o ) 2 ). Note that this reward function encourages forward horizontal motion and increase in the horizontal velocity, while penalizing large vertical deviations fromy o . We set y o = 1 which is a usual vertical position for our sea star model in stable locomotion given the problem parameters. The returnR t = P ∞ t ′ γ t ′ − t r t ′ is defined as the infinite horizon objective based on the sum of discounted future rewards, whereγ ∈ [0,1] is known as the discount factor; it determines the preference for immediate over future rewards. We setγ = 0.99 to make the agents foresighted. The goal is to arrive at an optimal set of parametersθ that maximizes the expected returnJ(π θ ) =E π P ∞ t=0 γ t r t for a dis- tribution of initial states. Here, the expectation is taken with respect to the distribution over trajectories π (τ ) induced jointly by the tube feet dynamics, viewed as a partially-observable Markov decision process, and the policy π θ (a t |o t ). One approach to solving this optimization problem is to use a policy gradient method that computes an estimate of the gradient∇ θ J for learning. Policy gradient methods are widely used to learn complex control tasks and are often regarded as the most effective reinforcement learning techniques, especially for robotics applications [137, 120, 57, 105, 106]. Here, we use a specific class of policy gradient methods, known as actor-critic methods [69, 38] where the agents learn simultaneously a policy (actor) and a value function (critic). We implement this method using the clipped advantage Prox- imal Policy Optimization (PPO) algorithm proposed in [113]. This algorithm ensures fast learning and robust performance by limiting the amount of change allowed for the policy within one update. More de- tail of the mathematical foundation for the RL algorithm used here and its implementations are provided in Appendix A. In order to capture the hierarchical and distributed model of the control architecture, we propose a RL structure as show in Fig. 4.1. Each tube foot here acts as an agent with the ability to choose an actiona n in response to its local observationso n . The same control policy is cloned among all the tube feet and trained 32 simultaneously. The state of the system evolves according to the physics of the environment as a results of the action of all the tube feet and their interaction with the body and the substrate. Once the state of the system has evolved one time step, a global reward signal is returned to all the agents informing them of how well the trained policies are performing. RL Agent 1 Policy RL Agent n Policy RL Agent N Policy Physics RL Environment: sea star body + tube feet + substrate Action Observation a n (t) Reward ... ... Action Observation Action Observation Figure 4.1: Hierarchical structure for learning cooperative locomotion in tube feet. The RL policy is cloned among all the agents. Every agent observes local sensory cues and samples an action in response to this local observation. The same collective reward function is returned to all the agents. 4.2 Learningthecontrolpolicies We train the RL agents to perform the two different control decisions described in Sec. 4.1, while providing them with a set of candidate mechasensory cues (see table 4.1). Note that these sensory cues are egocentric, namely, the tube feet can only sense its own local sensory cue. The cues are broadly categorized into axial cues that are measured longitudinally along the tube feet, and cues that are sensed in the shear direction. We perform all the trainings on two tube feet carrying a point mass on a flat horizontal substrate. As we show later, we gauge the performance of the RL policy by testing it on a rigid body carried by 100 tube feet. By training on only two tube feet, we significantly reduce our cost of training and probe the robustness and scalability of the trained policies to larger number of feet. For all three tasks, we globally prescribe the tube feet to step in the positivex-direction with a fixed recovery time τ =0. 33 Table 4.1: Candidate mechanosensory cues for control submodalities. Mechanosensing Candidatecues Axial stress / strain l l− l o F p F a Shear stress / strain θ sinθ F p sinθ F a sinθ Optimal mechanosensory cues For each one of the two decisions (detachment and push/pull), we provide all the cues to the agents, but in different training sets. The agents can only observe one cue in each training set and are never provided with both at the same time. And for every sensory cue provided we perform 10 training with different random seeds. In total, we perform 80 trainings for learning to detach and 80 trainings for learning to push/pull. The training results are shown in Fig. 4.2 (A-F). The top row (A-C) shows the results for learning when to detach and the middle row (D-F) shows the results for learning to push/pull. In Fig. 4.2(A, D), we show the reward gained by every training agent during training. The mechanosensory cue with the best performance in training is highlighted in orange. In panels (B, E), we calculate the average reward gained by each sensory cue defined as the accumulative reward over the whole training process, divided by the length of that training process in number of episodes. This is our metric of how training performance for the mechanosensory cues. The most effective mechanosensory cues for learning the decisions to detach and push/pull are axial strain (passive force) and shear strain, respectively. Optimal local control policy Next we delegate both tasks of deciding when to detach and when to push or pull to the RL agents simultaneously. Here we provide the tube feet with the axial strain and shear strain as sensory inputs, since they were the most effective for each learning each of the two decisions. Fig. 4.2(G-I) shows the training performance and the trained policies for the two actions. The colormaps depict the likelihood of performing each action in the observation space. The dashed lines show our designed control policies. We see some differences in the designed and RL-trained policies, especially for the decision to push/pull. 34 Episodes Reward 0 3 2 1 0 5 2 1 3 4 × 10 3 A Likelihood to detach 0 0.5 1 Episodes Reward 0 3 2 1 0 5 2 1 3 4 Designed policy Average RL policy −0.5 0 0.5 Axial strain, l − l o −1 −0.5 0 0.5 1 Shear strain, sin θ Likelihood to push 0 0.5 1 Reward 0 3 2 1 × 10 3 × 10 3 × 10 3 × 10 3 B C D E × 10 3 Episodes 0 5 2 1 3 4 Likelihood to push 1 0.5 0 Likelihood to detach 1 0.5 0 −1 −0.5 0 0.5 1 Axial strain, l − l o 0 −1 1 Shear strain, sin θ detach 0.5 −0.5 −1 −0.5 0 0.5 1 Axial strain, l − l o 0 −1 1 Shear strain, sin θ 0.5 −0.5 0.6 0.4 0.2 0 Average RL policy F I Average reward Mechanosensory cues l F p F a F a sinθ F p sinθ θ sinθ F p l F a F a sinθ F p sinθ θ sinθ 0.6 0.4 0.2 0 Average reward Mechanosensory cues Figure 4.2: Learning curves for learningA. the decision to detach andB. the decision to switch from active pushing to pulling. The bar plots show the ranking of the provided mechanosensory cues in terms of the average reward they gain during trainig. The orange curves in C. and D. show the control policy trained by RL using the most effective cue for the two learning tasks, respectively. The grey lines show our user-designed policies. E. Learning curve and F. the trained policy for learning both decisions of detachment and switching from pushing to pulling simultaneously. 4.3 Testingthecontrolpolicies To test the scalability of the control policy trained on 2 tube feet, we test it on sea stars equipped with 10 and 100 tube feet. To further gauge the robustness of the policies, we modify the tube feet model to not immediately reattach after detachment (τ > 0). Note that all of the results and trainigs thus far are done whenτ = 0. To this end, To this end, we consider tube feet can be in two states: (i) engaged with the substrate (power stroke) and (ii) detached (recovery stroke). In this model, tube feet transition from engaged to detached state in response to local sensory cues according to either the designed policy or the policies trained by RL. However, instead of reattaching to the substrate at the very next time step, now a 35 tube foot transitions back to the attached state randomly. Namely, the probability that at timet a detached tube footn transitions to the attached state is given by P n (t)= Z t to λdt, (4.1) whereλ is the rate of re-attachement andt o is the start of the detachment phase for that tube foot. Note that large values ofλ imply faster reattachment and shorter recovery stroke. If a tube foot transitions to the attached state, it takes a step of size∆ θ n in the direction of motion, then reattaches to the substrate. We test the RL policy trained on both actions of detaching and pushing/pulling simultaneously on a sea star of body of length 40 carried by 10 and 100 tube feet. Note that we choose this body length to be closer to the actual body to feet length ratio as reported in table 3.1. For a more fair comparison of the results, we use the same body length to test the body with 10 tube feet. The testing results for 10 tube feet are shown in Fig. 4.3. Fixing the body length, we explore the range of body weightmg ∈ (0.2,5) and rate of reattachmentλ ∈ (0.2,2), by performing 10 simulations (with different random initial conditions) for each value. We then report the average displacement of the body (for the cases where locomotion is successful), the success rate, and the average cost of transport per tube foot. As evident from the figure, the RL-trained policy is significantly more successful and robust relative to our designed policy give the same parameter values for ten tube feet. Same testing metrics are reported for 100 tube feet in Fig. 4.4. Interestingly, the success rate is very high in these range of parameters for both the designed and RL-trained policies. This implies that, for a small number of feet, the model performance is very dependent on the policy. However, as the number of feet increase the tube feet become more robust to variations in their control policy. 36 success rate displacement 0% 100% 50% 0.5 0.2 0.3 0.4 0.05 0 0.025 COT per foot RL policy designed policy rate of reattachment, λ 0.2 0.6 1 1.4 1.8 rate of reattachment, λ 0.2 0.6 1 1.4 1.8 1 2 3 4 5 Weight, mg 1 2 3 4 5 Weight, mg rate of reattachment, λ 0.2 0.6 1 1.4 1.8 A B 0.2 1 1.8 1 3 5 0.2 1 1.8 1 3 5 0.2 1 1.8 1 3 5 0.2 1 1.8 1 3 5 0.2 1 1.8 1 3 5 0.2 1 1.8 1 3 5 Figure 4.3: Testing performance of the control policies on 10 tube feet usingA. the user-designed policy andB. the RL-trained policy. From left to right, the colormaps show displacement, success rate and cost of transport for body weight. The dashed lines mark the contour lines for the specified values. The insets show a smaller range of the parameter space. success rate displacement 10 20 30 40 weight, mg 0% designed policy 2 4 6 8 10 rate of reattachment, λ 2 4 6 8 10 rate of reattachment, λ 2 4 6 8 10 rate of reattachment, λ 100% 50% 0% 2.2 1 1.4 1.8 0.05 0 0.25 COT per foot 10 20 30 40 weight, mg RL policy A B 0.04 0.04 2 2 Figure 4.4: Testing performance of the control policies on 100 tube feet usingA. the user-designed policy andB. the RL-trained policy. From left to right, the colormaps show displacement, success rate and cost of transport for body weight. The dashed lines mark the contour lines for the specified values. 37 4.4 Decentralizedcontrolwithlocaldirectionalitycommand Having found optimal sensory cues and control policies for locomotion using a global knowledge of direc- tionality, we now turn to the question of whether directionality can be implemented as a control decision in the local feedback loops. Namely, can each tube foot decide, based on local sensory cues only, whether to push or pull during the power stroke and which direction to step once in the recovery stroke? Note that these decisions need to be made regardless of any prior notion of whether we want the sea star to move to the left or right. To this end, we remove the connection marked by the green lines in Fig.?? and propose the following local control policy for every tube foot during the power stroke, Forl n <l max : |θ n (t)|−| θ n (t− ∆ t)|>0: push, |θ n (t)|−| θ n (t− ∆ t)|<0: pull. (4.2) where|θ n (t)| denotes the absolute value of the tilt angle and serves as an approximate for the magnitude of shear strain in every foot. According to Eq. 4.4, every foot needs to sense whether the shear strain magnitude has increased or decreased at its base since the previous timestep. If increasing (decreasing), then the foot must generate a pushing (pulling) force. 60 −60 0 20 40 −40 −20 horizontal displacement, x proportion of occurrences 0% 5% 10% 15% 20% unsuccessful locomotion successful locomotion to the right successful locomotion to the left Sea star body Motor control in response to local cues Mechanical coupling Tube feet Decentralized control model A Detach Attach Pull/Push Detach Attach Pull/Push Detach Attach Pull/Push Direction Direction Direction B Figure 4.5: Decentralized control of tube feet. A. Proposed architecture for a completely decentralized motor control of tube feet. The directionality command is added into the local control feedback loops. B. Symmetry-breaking from local control of tube feet. Histogram of the final horizontal position of a rigid body carried by 10 tube feet with a fixed recovery time τ =2 based on 1000 random simulations. In about80% of the cases, the tube feet can successfully carry the body to either left or right without any global directionality command. 38 Similar to the previous policies, every foot will complete its power stroke and enter the recovery stroke, once its maximum length has been reached. At the end of the recovery stroke (the duration of which can be a fixed or random variable), every foot decides its point of reattachment according to Eq. 3.5, where now∆ θ n is given by, − l ˙ θ n >0: e=+e x , − l ˙ θ n <0: e=− e x . (4.3) where− l ˙ θ n is the local rotational velocity at the base of thenth foot. Note that if in either of these two binary policies, the inequalities don’t hold, then the foot would take a random action from the two choices. 4.5 Discussion In this chapter, we used resolved optimal mechanosensory cues for cooperative transport in the context of our tube feet inspired and hierarchical control model in Chapter 3. Using a model-free RL approach, we showed that tube feet can most effectively learn to detach and decide when to actively push or pull if provided with their local axial and shear strains. We then used these two mechanosensory cues to train the tube feet the complete local feedback control loops. We found that the policy trained on a point mass to two tube feet can be generalized to ten and even a hundred tube feet. By training on only two feet and deploying the policy on more feet, we significantly save on the computational costs. Our findings shows that the policy trained by RL performs better than our user-designed policy when implemented on a rigid body with 10 feet. However, our user designed policy’s performance is on par with the RL policy if implemented on 100 feet. This implies that the robustness of the model to the control policy increases as the number of actuators in the model increase. 39 Next, we made the model more efficient by proposing a completely decentralized control mode. By removing the global directionality command and integrating it into the local feedback loops of each foot, we eliminated the cost associated with a global decision making center. We showed that our new controller leads to coherent locomotoin on tube feet reaching a consesnsu on their global direction of motion. This chapter is an on-going research work and will be updated soon with new results and figures. 40 PartII Self-organizationofflow-coupledswimmersintospatialpatterns 41 Chapter5 Introductionandmotivation Fish schools are ubiquitous in aquatic life, with half of the known fish species thought to exhibit schooling behavior during some phase of their life cycle [115]. A popular yet challenging to assess incentive is that schooling fish gain hydrodynamic advantage over solitary fish [101, 84, 124, 4]. The propulsive motion of a swimming fish creates alternating patterns of vorticity in water [80, 126, 93, 20, 72] that could in principle be exploited by neighbors to reduce energy expenditure by positioning themselves at locations that minimize drag [79, 14] or enhance thrust [114]. However, it is unclear which spatial patterns are energetically advantageous, and whether schooling fish actively seek energetically-favorable positions or if flow interactions create physical conditions that allow fish to self-organize passively into stable formations that lead to energy savings. Weihs [136] suggested that schooling fish maximize hydrodynamic benefits when assuming a lattice topology with repeated diamond pattern, and Lighthill [81] pondered whether such pattern is brought into play by passive hydrodynamic forces or whether elaborate control mechanisms are required to maintain it. Subsequent field and laboratory experiments [101, 84, 4] have shown that schools do not generally conform to highly regularized patterns [127, 97], and schooling fish dynamically change their position in the school. The aforementioned crystal lattice models do not capture the variability that fish exhibit in field 42 and laboratory experiments [101, 84], and the broader question of how flow interactions benefit schooling remains unresolved. The spatiotemporal motion of fish and wake tends to obscure our understanding of how schooling fish gain hydrodynamic benefits at intermediate to high Reynolds numbers and makes it difficult to dis- ambiguate between the effects of passive flow interactions [9, 67, 143, 60, 94, 70] and the need for active feedback control [133]. Recent studies addressed these challenges in pairwise formations because these formations are experimentally and computationally tractable with regard to the unsteady fluid-structure interactions [9, 67, 143, 60, 94, 70], and because they are biologically relevant due to their abundance in natural fish populations [78, 61, 40, 112]. Even in schools, fish tend to swim close to a single or few neighbours [100]. Key insights into how flow interactions affect self-organization came from the physical experiments of Newboltetal. [94], where they showed that pairs of freely-swimming hydrofoils, positioned in tandem and flapping at phase ϕ relative to each other, swim cohesively at a separation distanced linearly proportional toϕ . To crossover from a mechanistic view of flow physics to a biological hypothesis of how fish should behave to exploit hydrodynamic interactions, Li et al. [78] combined robotic and biological experiments. They showed, using a pair of identical robotic fish, that maximum energy savings occur when the relative tailbeat phase ϕ is linear with streamwise separation distance d. That is, they uncovered in a separate system the same linear relationship between phase and distance noted in [94] and associated this relation- ship with energy saving. From this, they extrapolated a strategy for how individual fish should behave in a school to maximize hydrodynamic benefits: the follower should match its tailbeat phase with the lo- cal vorticity created by the leader, which they called Vortex Phase Matching (VPM). Interestingly, pairs of freely swimming fish seemed to follow this rule even in the absence of visual and lateral line sensing. 43 While compelling, the distinct set-ups in [94, 78] obscure the connection between the physical and biological mechanisms leading to cohesive pairwise formations and do not clarify the limitations imposed by passive flow interactions or active vortex phase matching on schooling. In this part of the manuscript, we investigate the emergent dynamics, hydrodynamic benefits and stability of interacting swimmers in variable spatial and temporal formations. Specifically, in Chapter 6 we propose our mathematical model of flow-coupled flapping swimmers in the context of the vortex sheet model. In Chapter??, we study the emergent dynamics in a pair of flapping swimmer, and we analyze the experiments of [94, 78, 4, 110, 70] in light of existing [67, 103, 41, 70, 3] and our own mathematical models. Finally, in Chapter 8, we probe the emergent formations and the hydrodynamic benefits associated withe large groups of up to ten interacting swimmers and propose a novel control approach for collective motion in large unstable schools. 44 Chapter6 Vortexsheetmodelofflappingswimmers In our model, every swimmers is approximated as rigid plates, of finite chord length L, small thickness e ≪ L, undergoing pitching motion around its leading edge. We solve the fluid-structure interaction between the swimmer and the surrounding fluid using an inviscid vortex sheet model. In viscous fluids, when a flow passes across a solid surface, there forms a boundary layer, within which the fluid velocity has a gradient. On the upper boundary of the layer, the flow velocity is the same as uniform flow. On the lower boundary, due the no slip-boundary condition, the tangential velocity is equal to that of the solid boundary. The boundary layer is formed along the sides of the swimmer, and it is swept away at the swimmer’s tail to form a shear layer that rolls up into vortices. In the vortex sheet model, the swimmer is approximated by a bound vortex sheet, denoted byl b , whose strength ensures that no fluid flows through the rigid plate, and the separated shear layer is approximated by a free regularized vortex sheet l w at the trailing edge of the swimmer. The total shed circulationΓ in the vortex sheet is determined so as to satisfy the Kutta condition at the trailing edge, which is given in terms of the tangential velocity components above and below the bound sheet and ensures that the pressure jump across the sheet vanishes at the trailing edge. To express these concepts mathematically, it is convenient to use the complex notation z = x+iy, wherei = √ − 1 and(x,y) denote the components of an arbitrary point in the plane. The bound vortex sheetl b is described by its positionz b (s,t) and strengthγ (s,t), wheres∈ [0,L] denotes the arc length 45 along the sheetl b . The separated sheetl w is described by its positionz w (Γ ,t),Γ ∈[0,Γ w ] whereΓ is the Lagrangian circulation around the portion of the separated sheet between its free end in the spiral center and the pointz w (Γ ,t). The parameterΓ defines the vortex sheet strength γ =dΓ /ds. 6.1 Flowfield By linearity of the problem, the complex velocity w(z,t) = u(z,t)− iv(z,t) is a superposition of the contributions due to the bound and free vortex sheets w(z,t)=w b (z,t)+w w (z,t). (6.1) In practice, the free sheet l w is regularized using the vortex blob method to prevent the growth of the Kelvin-Helmholtz instability. The bound sheet l b is not regularized in order to preserve the invertibility of the map between the sheet strength and the normal velocity along the sheet. The velocity components w b (z,t) andw w (z,t) induced by the bound and free vortex sheets, respectively, are given by w b (z,t)= Z L 0 K o (z− z b (s,t))γ (s,t)ds, w w (z,t)= Z Γ w 0 K δ (z− z w (Γ ,t))dΓ , (6.2) whereK δ is the vortex blob kernel, with regularization parameterδ , K δ (z)= 1 2π i z |z| 2 +δ 2 , z =x− iy (6.3) Ifz is a point on the bound sheet for whichδ =0,w b is to be computed in the principal value sense. 46 Free vortex sheet Bound vortex sheet u + u − γ(s,t) Γ w e x e y z w (s,t) z b (s,t) F D n θ B A Figure 6.1: A. Schematic of the vortex sheet model for a two-dimensional flapping swimmer. B. Depiction of the different hydrodynamic forces acting on the swimmer. The position of the bound vortex sheet z b is determined from the plate’s flapping (y(t),θ (t)) and swimmingx(t) motions. The corresponding sheet strengthγ (s,t) is determined by imposing the no pen- etration boundary condition on the plate, together with conservation of total circulation. Let n(s,t) = − sinθ +icosθ be the upward normal to the plate, the no penetration boundary condition is given by Real[wn] z b = Real[w swimmer n], (6.4) where w swimmer = ˙ x− i˙ y− i ˙ θ [¯z b − (x− iy)]. (6.5) Conservation of the fluid circulation implies that if initially the circulation of the whole fluid domain is zero, then it is always zero, i.e., R l b γ (s,t)ds+Γ w (t)=0. The circulation parameterΓ along the free vortex sheetz w (Γ ,t) is determined by the circulation shed- ding rates ˙ Γ w , according to the Kutta condition, which states that the fluid velocity at the trailing edge is finite and tangent to the flyer. The Kutta condition can be obtained from the Euler equations by enforcing that, at the trailing edge, the difference in pressure across the swimmer is zero. To this end, we integrate the balance of momentum equation for inviscid planar flow along a closed contour containing the vortex sheet and trailing edge, [p] ∓ (s)=p − (s)− p + (s)=− dΓ( s,t) dt − 1 2 (u 2 − − u 2 + ), (6.6) 47 where Γ( s,t) = Γ w + R s 0 γ (s ′ ,t)ds ′ , 0 ≤ s ≤ L, is the circulation within the contour and p ∓ (s,t) and u ∓ (s,t) denote the limiting pressure and tangential slip velocities on both sides of the swimmer. Since the pressure difference across the free sheet is zero, it also vanishes at the trailing edge by continuity, which implies that ˙ Γ w =− 1 2 (u 2 − − u 2 + )| s=0 . (6.7) The values of u − and u + are obtained from the average tangential velocity component and from the velocity jump at the trailing edge, given by the sheet strength, evaluated ats=0 u= u + +u − 2 , u − − u + =γ 0 . (6.8) whereγ 0 is the bound vortex sheet strength at the trailing edge andu, also called the slip velocity, is the tangential velocity of the background flow relative to the training edge. Once shed, the vorticity in the free sheet moves with the flow. Thus the parameter Γ assigned to each particlez w (Γ ,t) is the value ofΓ w at the instant it is shed from the trailing edge. The evolution of the free vortex sheet z w is obtained by advecting it in time with the fluid velocity, ˙ ¯z w =w w (z w ,t)+w b (z w ,t). (6.9) 6.2 Forcesandmoments The hydrodynamic force acting on the swimmer due to the pressure difference across the thin flat plate is given by, Z l b n[p] ∓ ds=− F sinθ +iF cosθ, (6.10) 48 whereF = R l b [p] ∓ ds. The hydrodynamic moment acting on the swimmer about its leading edge is given by M = Real Z l b in(z le − z b )[p] ∓ ds , (6.11) wherez le is position of the leading edges=L. We introduce the drag force D that emulates the effect of skin friction due to fluid viscosity. This force is based on the Blasius laminar boundary layer theory, and was implemented by [30] in the context of the vortex sheet model. Blasius theory provides an empirical formula for skin friction on one side of a horizontal plate of length L placed in fluid of density ρ f and uniform velocity U. In dimensional form, Blasius formula isD =− 1 2 ρ f Lc f U 2 , where the skin friction coefficient C f = 0.664/ √ Re is given in terms of the Reynolds number Re = ρ f UL/µ . Substituting back in the empirical formula leads to D = − C d U 3/2 , where C d = 0.664 p ρ f µL . Following [30], we write a modified expression of the drag force for a swimming plate D =− C d (U 3/2 + +U 3/2 − ), (6.12) whereU ± are the spatially-averaged tangential fluid velocities on the upper and lower side of the plate, respectively, relative to the swimming velocityU, U ± (t)= 1 2l Z L 0 u ± (s,t)ds− U, (6.13) where u ∓ (s,t) denotes the tangential slip velocities on both sides of the plate. We estimate C d to be approximately0.02 in the experiments of [110]. 49 6.3 Numericalimplementation Each bound vortex sheet is discretized by2n+1 point vortices atz b (t) with strength∆Γ = γ ∆ s. These vortices are located at Chebyshev points that cluster at the two ends of the swimmer. Their strength is determined by enforcing no penetration at the midpoints between the vortices, together with conservation of circulation. The free vortex sheet is discretized by regularized point vortices atz w (t), that is released from the trailing edge at each timestep with circulation given by (6.7). The free point vortices move with the discretized fluid velocity while the bound vortices move with the swimmer’s velocity. The discretization of equations (7.1) and (6.7, 6.9) yields a coupled system of ordinary differential evolution equations for the swimmer’s position, the shed circulation, and the free vorticity, that is integrated in time using the 4th order Runge-Kutta scheme. The details of the shedding algorithm are given in [95]. The numerical values of the timestep∆ t, the number of bound vorticesn, and the regularization parameterδ are chosen so that the solution changes little under further refinement. Dissipationtime To emulate the effect of viscosity, we allow the shed vortex sheets to decay gradually by dissipating each incremental point vortex after a finite time T diss from the time it is shed into the fluid. LargerT diss implies that the vortices stay in the fluid for longer times, mimicking the effect of lower fluid viscosity. To investigate the effect of the dissipation time on the forces acting on the plate, we consider the problem of a flat plate fixed at one end in an oncoming uniform flow. A detailed study of dissipation time is available in [huang2018]. Fig. 6.2 shows the magnitude of the total hydrodynamic force as a function ofT diss for a flat plate held at a fixed angle of attack π/ 12. For a given value ofT diss , there is an effective ’dissipation force’ that is equal to the difference between the value of F as T diss → ∞ and the value of F at T diss . While F increases as T diss increases, the dissipation force decreases, in agreement with physical intuition. In figure 6.3, we consider the aerodynamic force acting on a plate undergoing pitching 50 0.4 0.6 0.8 1 0 1 2 3 4 5 6 7 8 9 Fixed plate U Figure 6.2: Force magnitude versus dissipation timeT diss for a fixed plate at angle π/ 12 in an oncoming uniform flow U∞ =− 1. oscillations in uniform flow. Figure 13(a) shows the force magnitude versus time for oscillation amplitude π/ 12 , frequency 1 andτ diss = 1.5. The total integration time is equivalent to fifty cycles of oscillations. Figure 6.3(b) shows the force magnitude averaged over the last ten oscillation cycles versus T diss . The force behavior is more complicated than that of the fixed plate: the force increases as τ diss → nT , where T =1 is the period of oscillation of the plate andn is an integer. The reason is that atτ diss =nT the force induced byτ diss is in resonance with the frequency of oscillation of the plate. Therefore,τ diss /T should 50 100 150 200 250 300 350 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 0 40 42 44 46 48 50 32 34 36 38 30 -200 -100 0 100 200 Flapping plate U A B Figure 6.3: A pitching plate in an oncoming uniform flow U =− 1. A. Force magnitude versus time forT diss = 0.5 and B. average force versus dissipation timeT diss . The amplitude of force is averaged over the last ten cycles of oscillation, highlighted in grey inA.. The plate’s oscillation amplitude is equal toπ/ 12 and frequency is equal to 1. not be an integer value. It is important to note that forτ diss away from the resonant conditions, the total force follows a similar trend to that observed in figure 6.3(b) in that the force increases as the dissipation time increases, as highlighted by the red dashed line. 51 We refer the reader to [47] for a detailed analysis of the effect of dissipation time on the hydrodynamic forces on a stationary and moving plate in the vortex sheet model. Details of the numerical validation in comparison to [54] and [55] are provided in [46]. 52 Chapter7 Self-organizationinpairsofflappingswimmers Inspired by the experiments of [94, 78] (Fig. 7.1A), we study self-organization in the context of a pair of flapping swimmers, coupled passively via the fluid medium, with no mechanisms for visual [89] and flow sensing [111, 23],or feedback control [77]. We first introduce, the problem formulation in the context of the vortex sheet model introduced in Chapter 6. Then, we solve the pairwise interaction in freely swimming and tethered pairs of swimmers at different lateral offsets and phase lags. Our results confirm the previous findings that mechanical coupling via the fluid medium only, with no sensory feedback, self-organize into stable, energy saving formations. We further show, and provide a mechanistic explanation for, the existence of a universal relationship where, in these stable formations, the separation distance between pairs of swimmers varies linearly with the difference in flapping phase. 7.1 Mathematicalmodelingofflow-coupledswimmers The swimmers are rigid, of finite chord length L, small thicknesse≪ l, and mass per unit depthm, and undergo pitching oscillations of identical amplitudeA and frequencyf in the(x,y)-plane of motion, such that the pitching angle for swimmerj, is given byθ j = Asin(2πft +ϕ j ). We setϕ 1 = 0 andϕ 2 =− ϕ , where ϕ is the phase difference between the oscillating pair. We fix the lateral distance ℓ between the swimmers to lie in the range ℓ ∈ [− L,L] (Fig. 7.1A), and we allow the swimmers to move freely in the 53 x-direction in an unbounded two-dimensional fluid domain of density ρ . Hereafter, we scale all parameters using the bodylength L as the characteristic length scale, flapping period T = 1/f as the characteristic time scale, and ρL 2 as the characteristic mass per unit depth. Accordingly, velocities are scaled by Lf, forces byρf 2 L 3 , moments byρf 2 L 4 , and power byρf 3 L 4 . The equations governing the free motionx j (t) of swimmerj are given by Newton’s second law (down- stream direction is positive) m¨x j =− F j sinθ j +D j cosθ j , (7.1) where the hydrodynamic forces acting on swimmerj is decomposed into a pressure forceF j acting in the direction normal to the swimmer and a viscous drag forceD j acting tangentially to the swimmer. These forces depend on the instantaneous state of the swimmers and their time history. To compute the hydrodynamic forces and swimmers motion, we solve the fluid-structure interactions (FSI) using the vortex sheet model, detailed in Chapter 6 and a computational fluid dynamics (CFD) solver of the Navier-Stokes equations based on an adaptive mesh implementation of the immersed boundary method [48], (Appendix ??). Due to its computational efficiency, we use the vortex sheet model more extensively to explore the large parameter spaces and generalize the problem to larger groups of swimmers. 7.2 Universalspacing-phaserelationshipinemergentpairwiseformations Starting from the case of inline swimmers ℓ = 0 considered in [9, 94], we find that the swimmers self- organize into relative equilibria where the swimmers arrange themselves at a streamwise separation dis- tance d(t) that is constant on average, and the two swim together as a single formation at an average free-swimming speedU that is close to the free-swimming speed of the corresponding solitary swimmer (typically within 10%). 54 When the two swimmers are initially placed side-by-side at a lateral offset ℓ, they reach a relative equilibrium where they travel together at a close, but non-zero, average spacingd, implying that a perfect side-by-side configuration is unstable but the more commonly-observed configuration [78] where the two swimmers are slightly shifted relative to each other is stable. Hereafter, using the notation introduced in [4], we refer to this tightly-spaced, nearly side-by-side configuration as the phalanx formation. These emergent equilibria are not unique. Increasing d(0) causes the swimmers to reach different equilibrium formations at larger separation distance d. These observations are consistent between the CFD and VS simulations and recapitulate previous findings [ Peng2018, 143, 9, 110, 99, 24, 41, 3]: for inline and inphase swimmers, the average spacingd at these equilibria is approximately an integer multiple of UT (Fig. 7.1D), as noted in [9, 110, 94, 41]. Interestingly, when offsetting the swimmers laterally, these equilibrium formations persist, in addition to the phalanx formation atd/UT ≈ 0 (Fig. 7.1D). In all our CFD simulations, the Reynolds number Re =ρUL/µ based on the emergent swimming speed, and that based on the tailbeat velocity Re =2πρAfL/µ , are≈ 2000 and1600 which is of the same order of magnitude as in the experiments of [9, 94] and an order of magnitude larger than the CFD simulations in [9, 3]. We next examined the effect of phase ϕ on the emergent traveling formations. In the third panel of Fig. 7.1D, we setℓ = 0 and fixed the initial conditions so as to settle on the first equilibrium d/UT ≈ 1 when ϕ = 0. In both the CFD and VS simulations, the spacing d/UT at equilibrium increased with increasingϕ in a linear manner. This linear spacing-phase relationship is apparent in Fig. 7.2A where we plot the emergent average spacingd/UT as a function ofϕ for various values ofℓ. To probe the universality of this linear spacing-phase relationship, we compile, in addition to our CFD and vortex sheet results, a set of experimental and numerical data from the literature [67, 94, 78, 41, 70] (see ??. Data are superimposed on Fig. 7.2A and includes CFD simulations of deformable flapping flags (⋆) [67] and flexible airfoil with low aspect ratio ( ▷) [3], physical experiments with heaving (⃝ ) [94] 55 ×(UL) Li et al. 2020 A 0.5 1 1.5 2 increasing φ 30 20 10 0 5 15 25 Time, t/T 0 Spacing, d /UT VS model CFD model B Vortex sheet (VS) model 0.01 −0.01 0 Phalanx Inline VS 1 2 1 2 Power saving CFD 1 2 VS CFD Phalanx 1 2 0% 50% 100% Newbolt et al. 2019 CFD model Inline ×(U/L) 10 -10 0 increasing 30 20 10 0 5 15 25 Time, t/T 30 20 10 0 5 15 25 Time, t/T E Inline C D 2.5 vorticity circulation Phalanx Figure 7.1: Flow-coupled oscillators self-organize into stable and energetically favorable pairwise formations. A. Schematic of existing experimental work [94, 78] and B., C. our two proposed mathematical models of flow-coupled oscillators for inline and phalanx schooling. D. Time evolution of streamwise distance for a pair of freely swimming oscillators in the vortex sheet and CFD models. From left to right, the first panel: more than one equilibrium distance emerge for a pair of in-phase and in-line oscillators by varying the initial distance. The second panel: for a pair of in-phase swimmers the equilibrium distances slightly vary as we increase the lateral spacing between the pair. The third panel: for a pair of in-line oscillators and for the same initial condition, the equilibrium distances increase as we increase the follower’s phase lag. E. The distribution of power saving among the leader and follower for inphase Phalanx and inline pairs in the CFD and vortex sheet models. In all of our simulations, the amplitude and frequency of pitching areA=15 ◦ andf =1, respectively. 56 average power change per swimmer A Separation distance, d /UT 0 1 2 1.5 0.5 Li et al. 2020 Newbolt et al. 2019 CFD B VS CFD VS power saving power waste -50% 0% Kurt et al. 2021 Kim et al. 2010 0 1 2 1.5 0.5 Separation distance, d /UT Phase lag, φ/2̟ 0 0.25 0.5 0.75 1 2.5 2.5 Inline Phalanx Peng et al. 2018 Ramananarivo et al. 2016 Heydari et al. 2020 50% Arranz et al. 2022 Figure 7.2: A. The emergent relative equilibria in a pair of in-line and Phalanx oscillators in our vortex sheet model and CFD simulations (Reynolds number≈ 2000) are consistent with vortex phase matching formations observed in existing experimental and numerical studies [94, 78, 70, 67, 103, 41, 110, 3]. B. These stable formations save power (P single − P pair)/P single , where P pair =(P1 +P2)/2, compared to swimming alone in both CFD and vortex sheet models. and pitching (▽) [70] rigid hydrofoils in rotational and translational tank geometries, and biological data (△) extracted from pairs of live fish [78]. All data collapse onto the linear spacing-phase relationship d/UT ∝ ϕ/ (2π ), with largest variability exhibited by the biological data, where it is difficult to have a strictly constant value of the amplitude and frequency of flapping (Fig. 7.2A). These findings strongly indicate that flow-coupled, free-swimming oscillators self-organize into traveling equilibrium formations with linear spacing-phase relationship. This relationship is independent of the geometric layout (in-line versus laterally-offset swimmers), flapping kinematics (heaving versus pitching), material properties (rigid versus flexible), tank geometry (rotational versus translational), fidelity of the fluid model (CFD versus VS), and system (biological versus robotic). Observations that are robust across such a broad range of systems are expected to have common physical and mechanistic roots that transcend the particular set-up or system realization. Importantly, these findings suggest that the VPM introduced in [78] as a strategy by which fish maximize hydrodynamic benefits may not be an actively-implemented strategy, but an outcome of flow interactions. We return to this point later. To evaluate the hydrodynamic advantages associated with these emergent formations, we compute the hydrodynamic powerP single of a solitary swimmer, as well as the hydrodynamic powerP j of swimmer j and 57 the average power P = ( P N j P j )/N per swimmer in formation. See B for details on the computations of power. The colormap in Fig. 7.2B represents the average hydrodynamic power saving per swimmer compared to swimming alone,(P single − P)/P single . The results show that, for alld (and thusϕ ), in both the CFD and VS simulations, the swimmers traveling in equilibrium formation save power compared to swimming alone. For inline formations, these hydrodynamic benefits are bestowed entirely on the follower, whose power savings can be as high as 60% compared to swimming alone (Fig. 7.2A,B) [41]. Interestingly, for the phalanx formation, the power savings, while slightly more modest, are shared equally between both swimmers. The bias in power savings between leader and follower for inline swimmers could be a contributing factor to the dynamic nature of fish schools [121, 109, 90], whereas the egalitarian distribution of benefits in the phalanx formation could explain the abundance of this configuration in natural fish populations [78]. 7.3 Insightsintopowersavingsandfluidforcesfromtetheredformations To further evaluate these energetic benefits, we tether the inline swimmers at a fixed distance d in oncom- ing flow as done in the robotic experiments of [78]. We set the flow speed U to be equal to the speed of a freely swimming solitary swimmer and we systematically evaluate the hydrodynamic power as a function of the fixed separation distance d/UT and flapping phase lag ϕ (Fig. 7.3A). The downstream swimmer interacts either constructively or destructively with the wake shed by the upstream swimmer, as noted in [67, 9, 78]. Importantly, constructive interactions and maximal energy savings occur along the (dashed black) lines that correspond to equilibrium configurations of the freely swimming pair (Fig. 7.2B). In Fig. 7.3B, we report the hydrodynamic forces acting on the follower over the space of phase lag and separation distance; black arrows represent the direction and magnitude of the force and the colormap is used to further highlight their properties: thrust forces are shown in red and drag forces in blue. The equilibrium lines in Fig. 7.2A are “attractors". At these lines, the hydrodynamic force is zero on average. 58 When the follower is tethered behind or ahead of an equilibrium line, it experiences hydrodynamic thrust or drag pointing towards that line. A closer examination of the results in Fig. 7.3B points to the existence of “repelling" lines (dashed grey lines) where the hydrodynamic force is also zero on average but near which the hydrodynamic forces point away from these lines. To further illustrate these ideas, we plot in Fig. 7.3C the hydrodynamic forces versus the separation dis- tanced/UT for inphase flapping (colored markers) and fitted them with a best sinusoidal curve (solid line). At the separation distanced/UT ≈ 1 andd/UT ≈ 2, which correspond to stable traveling equilibria of the pairwise formation, the follower is subject to zero hydrodynamic force, and a finite perturbation about these configurations in either direction generates restorative hydrodynamic forces (forces that act in the opposite direction to the imposed perturbation). At the separation distanced/UT ≈ 0.4 andd/UT ≈ 1.5, the hydrodynamic forces are also zero but a perturbation about these configurations generates repelling forces. These configurations correspond to unstable equilibria (not accessible in forward time-stepping) of the freely swimming pair, and play an important role in delineating the basins of attraction of the sta- ble equilibria. A direct comparison to the emergent formations in freely swimming pairs indicates that the separation distance d/UT ≈ 0.4 marks the boundary between the basins of attraction of the stable phalanx (d/UT ≈ 0) and first inline ( d/UT ≈ 1) equilibria, whereas the separation distanced/UT ≈ 1.5 marks the boundary between the basins of attraction of the first ( d/UT ≈ 1) and second (d/UT ≈ 2) inline equilibria; as highlighted in dashed lines in Fig. 7.1D). This analysis of tethered swimmers provides explanatory flow-centric mechanisms for (i) the existence of the traveling equilibria in pairs of free swimmers at separation distance where the net hydrodynamic force is zero, (ii) the local stability of these equilibria via restorative hydrodynamic forces in the vicinity of these equilibria, and (iii) their global basins of attraction, delineated by locations where the hydrodynamic forces also vanish, but around which these forces are repelling. 59 Separation distance, d /UT 0.5 1.5 2 1 0 −1.5 Fluid force on follower, 0 1 1.5 Phase lag, φ/2̟ 0 0.25 0.5 0.75 1 1.2 −1.2 0 hydrodynamic force on follower A B average power change per swimmer power saving power expenditure Phase lag, φ/2̟ 0 0.25 0.5 0.75 1 C 50% -50% 0% d U drag thurst Separation distance, d /UT 1 2 1.5 0.5 × 0.5 −1 −0.5 Figure 7.3: A. Average power expenditure (relative to a single swimmer) and B. hydrodynamic force on the follower in a pair of tethered inline swimmers as a function of phase lag and separation distance. The dark dashed lines denote the emergent formations in swimming pairs. A. Fluid force on the follower in a tethered pair of inline and inphase oscillators (shown by□ ) versus separation distance and a Fourier curve fit to the data (shown by the grey line). There are two stable fixed points around d/UT = 1,2. The fluid force act as restoring force that push and pull the follower back to the emergent equilibria distances in freely swimming pairs shown in Fig. 7.3 (and marked by the dashed lines). 7.4 Parametric analysis over the entire space of phase lags and lateral offsets Having demonstrated consistency in the emergence of flow-mediated stable equilibria in both the CFD and VS simulations, we now exploit the computational efficiency of the VS model to systematically investigate the emergence of traveling formations over the entire space of phase lag ϕ ∈ [0,2π ] and lateral offset ℓ∈[− L,L]. Equilibrium configurations are dense over the entire range of parameters: for any combination of phase lag ϕ and lateral distance ℓ, there exists an emergent equilibrium configuration where the pair of swimmers travel together at a separation distance d/UT (Fig. 7.4A). Perturbing one or both parameters causes the swimmers to stably and smoothly transition to another equilibrium at different spacing d/UT . Importantly, increasing the phase lagϕ shifts the equilibrium positions in the streamwise direction such that d/UT depends linearly on ϕ , but the effect of lateral distance for ℓ ≤ L is nonlinear and small: increasing the lateral offset ℓ by an entire bodylengthL changes the pairwise distanced/UT by at most 15%. 60 2.5 2 1.5 1 0.5 0 separation distance, d /UT A phase lag φ lateral distance , l 1 0 −1 −0.5 0.5 stability / cohesion B 2.5 2 1.5 1 0.5 0 separation distance, d /UT 2.5 2 1.5 1 0.5 0 separation distance, d /UT 0 ̟ C 30% 10% 20% power saving power expenditure 50% -50% 0% 2̟ 0.3 0 0.15 20% 20% Inline Phalanx average power change per swimmer × Figure 7.4: Pairwise formation of interacting oscillators at different phase lags and lateral spacing. A. Equilibrium separation distances between the oscillators, B. power saving, and C. slope of the restoring fluid forces on a perturbed follower (right) as a function of phase lag and lateral spacing in a pair of interacting oscillators. For all the possible phase lags and all the shown lateral spacing, the pair reach equilibrium formations that are stable and power saving relative to a single swimmer. The contour lines show where the pairwise interactions lead to10%,20% and30% power saving. We assessed the hydrodynamic advantages of these emergent formations by calculating the average hydrodynamic power per swimmer as in Fig. 7.3A. The average power expenditure per swimmer in these pairwise formations is always less than that of a solitary swimmer (Fig. 7.4B). The power saving varies as a function of both phase lagϕ and lateral distanceℓ. Contour lines enclosing equilibria characterized by more than10%,20% and30% power savings are superimposed on Fig. 7.4B. For the entire range of phase lag from0 to2π , one gets consistently over20% power reduction, as long as the the lateral offset is l≤ 0.25L. However, increasingℓ from0.25L toL reduces significantly the hydrodynamic benefit (to less than 10% power saving). That is, swimmers can take great liberty in changing their phase without compromising much the average energy savings of the school, as long as they maintain close lateral distance. To quantify the stability and cohesion of these traveling equilibria, we perturbed each equilibrium by displacing the follower in the positive and negative x-directions by δd = δx = 0.5L away from the equilibrium spacing d, and computed the subsequent time evolution of the separation distance d and x- component of the hydrodynamic forceF with the aim of quantifying variations in forceδF as a function of variations in distance from the equilibriumδd (SI, Fig. S7A). We applied this numerical stability analysis 61 to each equilibrium in Fig. 7.4A. The force-displacement response near each equilibrium exhibited the basic features of a linear spring-mass system (SI, Fig. S7B): for all emergent equilibria,δF/δd is negative, indicating that the hydrodynamic force acts as a restoring spring force that causes the initial perturbation to decay and that stabilizes the follower at its equilibrium position in the leader’s wake. Larger value of|δF/δd | implies faster convergence to the stable equilibrium and thus stronger cohesion. An analogy can be drawn between|δF/δd | and the slope of the fluid force in Fig. 7.3C, where negative values imply stability or restoring forces and positive values imply instability or repelling forces. In sum, all emergent equilibria reported in Fig. 7.4A are stable. Variations in phase lagϕ over the entire range fromϕ = 0 toϕ = 2π has minimal effect on the energy savings and cohesion of the pairwise for- mation for tight lateral distances,ℓ≤ 0.25L. However, both energy saving and cohesion drop noticeably as the lateral distanceℓ increases. The drop is especially drastic forl > 0.25L, where the follower is not strongly interacting with the wake of the leader. 7.5 Opportunitiesinthewakeofasolitaryswimmer Our results strongly imply that the hydrodynamic mechanisms that mediate power savings and cohesion in pairwise formations are universal. Here, we hypothesize that these formations, and their hydrodynamic properties, can be predicted entirely from kinematic considerations of the leader’s wake and follower’s oscillations with no consideration of the two-way coupling between the swimmers. To challenge this proposition, we examine the wake of a solitary swimmer in CFD and VS simulations (Fig. 7.7A), and we treat the follower as a particle located at a point (x o ,y o ) and undergoing prescribed oscillationsy(t) = y o +Asin(2πt − ϕ ). The wake is thus blind to the existence of the follower and the follower doesn’t interact with the flow. This model is analogous but not identical to the minimal model introduced in [9, 94]. The latter treats both leader and follower as particles with minimal ‘wakes’ and considers two-way coupling between them (Fig. 7.7B and SI, Fig. S.4). In our analysis, the leader and its 62 wake can be described to any desired degree of fidelity to the fluid motion, including using experimentally constructed flows when available. We define a flowagreementparameter V(x o ,y o ,ϕ,t ) that describes how well the oscillatory motion of the follower particle, with velocity vectorv(x o ,y o ,ϕ,t ) = 2πA cos(2πt + ϕ )e y , where e y is a unit vector in the y-direction, matches the local flow u(x o ,y o ,t) generated by the solitary swimmer V(x o ,y o ,ϕ )= 1 T R t+T t v· u dt ′ 1 T R t+T t v· v dt ′ . (7.2) The time instantt is chosen after steady state is reached. Positive (negative) values ofV indicate that the flow at (x o ,y o ) is favorable (unfavorable) to the follower’s flapping motion. In Fig. 7.5(A), we show a snapshot of the flow field behind a soliary swimmer in the CFD and VS models. In Fig. 7.5(B, C) We show the flow agreement parameter as a field over the physical space for ϕ = 0,π . Blue regions indicate where the local flow is favorable to the follower’s flapping motion. In both CFD and VS simulations, the locations in the wake of the solitary swimmer with the maximum flow agreement parameter closely coincide with the stable equilibria found by solving the coupled dynamics of the pairwise interactions (equilibria in two-way coupled models are marked by circles in Fig. 7.5(B, C)). These findings suggest a simple rule for identifying the locations of stable equilibria in two-way coupled models of pairs of swimmers: they correspond to locations(x o ,y o ) ∗ of maximum flow agreement parameters V(x o ,y o ,ϕ ). Next, we propose to evaluate the stability and cohesion of pairwise formations directly from the wake of the solitary swimmer. To this end, we define a thrust parameter T(x o ,y o ,ϕ ), knowing that the magni- tude of thrust of a flapping swimmer scales with the square of the swimmer’s lateral velocity relative to the surrounding fluid’s velocity [125, 31, 94], T(x o ,y o ,ϕ )=− 1 T R t+T t ((v− u).e y ) 2 dt ′ 1 T R t+T t (2v.e y ) 2 dt ′ . (7.3) 63 Figure 7.5: A. Snapshot of the vorticity and flow field in the wake of a solitary swimmer in our CFD and VS models. B. Flow agreement parameter field in the wake of a solitary swimmer and C. thrust parameter at lateral distancesℓ/L =0,0.5,1 for an inphase followerϕ =0. D. Flow agreement parameter field in the wake of a solitary swimmer and E. thrust parameter at lateral distancesℓ/L=0,0.5,1 for an antiphase followerϕ =π . 64 2.5 2 1.5 1 0.5 0 Separation distance, d /UT A phase lag φ 1 0 −1 −0.5 0.5 cohesion parameter B 2.5 2 1.5 1 0.5 0 Separation distance, d /UT 2.5 2 1.5 1 0.5 0 Separation distance, d /UT 0 ̟ C flow agreement parameter 1.2 −1.2 0 2̟ −1 0 −0.5 Lateral distance , Figure 7.6: A. Locations of maximum flow agreement parameter, B. the magnitude of flow agreement parameter and C. cohesion parameter at positions of maximum flow agreement parameter for different phase lags and lateral distance in the wake of a solitary swimmer. In Fig. 7.5(D, E), we plot, as a function of the streamwise separation distance d/UT , the value ofT for y o = ℓ = 0,0.5L,L for ϕ = 0,π . The thrust parameter at the locations (x o ,y o ) ∗ of the maxima of V(x o ,y o ,ϕ ) is minimum (nearly zero); the slope∂T/∂d is an indicator of linear stability, or cohesion, of these equilibria: an increase (or decrease) in distance is accompanied by an increase (decrease) ofT in the opposite direction, consistent with the negative slope of the hydrodynamic restoring forceδF/δd shown in Fig. 7.7D. Furthermore, we can explore the whole parameter space of phase lag and lateral offset, by computing the locations of maximum flow agreement parameter, its value and the corresponding cohesion parameter at those locations as shown in Fig. 7.6. The results closely resemble those shown in Fig. 7.4 from solving the pairwise dynamics. In Fig. 7.7A, we show, as a function of the follower’sϕ , the locations of the local maxima(x o ,y o ) ∗ of V(x o ,y o ,ϕ ) computed based on the CFD (♢ ) and VS (△) models. The distance to these maxima is scaled byUT , whereU here is taken to be the speed of the solitary swimmer. These maxima shift longitudinally by about 0.5UT as ϕ changes from ϕ = 0 to ϕ = π We superimpose onto these results the equilibrium configurations obtained from pairwise interactions in the context of the CFD ( ♦ ), VS (□ ), and time-delay 65 VS isolated swimmer 1 0.8 0.6 0.4 0.2 0 1 0 −1 −0.5 0.5 1 0.75 0.5 0.25 0 Normalized power saving Phase lag, φ / 2̟ 1 0 2 0.5 1.5 2.5 1 0.8 0.6 0.4 0.2 0 Normalized pairwise cohesion 1 0 −1 −0.5 0.5 Separation distance, d /UT A B C CFD isolated swimmer lateral distance , lateral distance , CFD pairwise interactions VS pairwise interactions time-delay particle model Figure 7.7: A. Location of of maximum flow agreement parameter as a function of flapping phase ϕ of a particle follower located in the wake of a solitary swimmer based on CFD (⋄ ) and VS (△) simulations. Separation distance as a function of phase based on pairwise interactions in CFD (⋄ ) and VS (□ ) models are superimposed. B. pairwise cohesion from the vortex sheet model and the slope of the thrust parameter from the reduced order model andF. power saving from the vortex sheet model and flow agreement parameter from the reduced-order model and versus lateral distance for in-phase swimmers. All the data points inB. andC. are normalized by their corresponding maximum values for comparison. particle (⃝ ) models, where we modified the latter to account for non-zero lateral offset (SI, Fig.S4). Predic- tions of the equilibrium configurations based on maximal flow agreement parameter agree remarkably well with actual equilibria based on pairwise interactions: they all follow the universal linear spacing-phase relationship shown in Fig. 7.1G. In Fig. 7.7(B, C) we plot∂T/∂d andV as a function of lateral distanceℓ/L forϕ = 0, both evaluated at the locations of maximalV in the wake of the solitary swimmer. We superimpose, respectively, the negative gradient of the restoring force− δF/δd and the power saving based on pairwise interactions of inphase swimmers using the CFD, VS, and time-delay particle models. All quantities are normalized by the corresponding maximal value to highlight variation of these quantities withℓ, as opposed to their absolute values. Pairwise cohesion and power savings are almost constant forℓ<0.25L, but decrease sharply asℓ increases. This trend is consistent across all models. 66 7.6 Discussion Our analysis of the free swimming of pairs of flapping swimmers underlines the role of the fluid medium in the cohesive and coordinated motion of schooling fish. Our findings are consistent with the current exper- imental and numerical literature on pairs of freely swimming and interacting swimmers where the spacing between the pair and their phase difference follows a linear relationship. We showed that the pair save significant amount of power relative to swimming alone while following this relationship. We compiled, in addition to our CFD and vortex sheet results, a set of experimental and numerical data from the liter- ature [67, 94, 78, 41, 70] where the linear phase-spacing relationship was reported. The consistency of all these datasets show the universality of this relationship, which seems to be independent of flow properties (Reynolds number) and the swimmers body shape or flapping motion. By preforming numerical simula- tions on pairs of tethered swimmers in oncoming flow, we provided a mechanistic explanation for this universal linear relationship. Namely, we found that in tethered pairs, the downstream swimmer interacts either constructively or destructively with the wake shed by the upstream swimmer, as shown in Fig. 7.3 and noted in [67, 9]. Furthermore, we looked at the flow field generated by a solitary swimmer and found positions in the wake with that provide the follower with opportunities of power saving. We characterized this opportunity as the flow agreement parameter and showed that constructive and destructive interac- tions between the follower and the wake correspond to positive and negative flow agreement parameter. Finally, we showed that the wake of a solitary swimmer can be used to make accurate predictions of the equilibria in interacting pairs. Namely, we found that the stability and cohesion of the equilibria in freely swimming pairs correlate with variation of the thrust parameter at the location of these local maxima in flow agreement parameter, and power savings correlate with the maximum value of the flow agreement parameter. 67 Chapter8 Emergentformationsinlargegroupsofswimmers So far, our study and the majority of the existing literature, has been focused on self-organization of swim- mers in the context of pairwise interactions only. Next, we ask how do these insights from the pairwise formations scale to larger groups. To address this question, we study the freely-swimming dynamics of groups with 3 and 4 swimmers and later extend to groups with 6 swimmers. We also compare predictions from the flow agreement parameter with emergent behavior from the flow-coupled dynamics in larger groups of swimmers. 8.1 Stabilityoflargegroups First, we model the free swimming of a group consisting of 3 and 4 inline swimmers. We calculate the agreementV(x o ,y o ,ϕ =0) using the flow field generated by the emergent formation of two inline swim- mers flapping inphase. The location of maximum V correctly predicts the existence of an equilibrium formation with three inline swimmers, which we confirm by solving for the coupled dynamics of three swimmers (Fig. 8.1A). Flow interactions lead to a stable emergent formation where swimmer 3 experiences the largest hydrodynamic advantage (up to 120% power saving!) and highest flow agreement, swimmer 2 receives moderate benefits (65% power saving), and swimmer 1 no benefit at all (Fig. 8.1E). 68 Given the emergent formation of three inline swimmers, we calculate the corresponding flow agree- ment parameter V(x o ,y o ,ϕ = 0) to predict where a trailing swimmer should position itself. Positive agreement in the (blue) region behind the third swimmer is weak, indicating weaker potential for energy saving by a fourth swimmer. A direct comparison of the flow agreement field of two versus three inline swimmers shows that it does not scale with the number of inline swimmers (Fig. 8.1A). Further, by compar- ing the period-average flow field in the x-direction in Fig. 8.1B, we find a stronger jet in x-direction behind the three inline swimmers. Indeed, by solving for the free dynamics of four inline swimmers, we find that the leading three swimmers maintain cohesion, at hydrodynamic benefits similar to a formation of three, but swimmer 4 separates from the rest of the group. These findings demonstrate that flow interactions alone are not sufficient to maintain inline formations in inline groups with more than three swimmers. Stabilityviapassiveformations Next, to resolve the stability issue we investigate whether an unstable train of inline swimmers can self-organize into a single formation by alternating the lateral offset ℓ=0.25L between consecutive swimmers (Fig. 8.1A). Such staggered patterns are commonly observed in groups of flying birds [109]. By staggering the leading three swimmers, the trailing (red) region with negative flow agreement is slightly weakened and the subsequent (blue) region of positive flow agreement is slightly strengthened compared to the inline formation such that, when solving for the coupled dynamics, swimmer 4 is able to maintain its position in formation (Fig. 8.1A), albeit at no power savings compared to solitary swimming (Fig. 8.1E). By staggering the swimmers, swimmer 4 is positioned in the region with weaker jet in the x-direction, enabling it to maintain its position in formation (Fig. 8.1A). Its low flow agreement parameter is consistent its negligible power savings relative to a solitary swimmer. The staggered pattern proposed in Fig. 8.1A is not unique for maintaining cohesion. Other opportuni- ties exist for the swimmers to position themselves beneficially relative to each other. In Fig. 8.2, we show four such arrangements. When swimmers 1 and 2 pair up to swim in the lead, swimmer 4 collides with swimmer 3. If, however, swimmer 4 is positioned at the subsequent equilibrium behind swimmer 3, all 69 A separation φ = 0 φ = 0 φ = 0 1 3 2 4 1 3 2 4 1 3 2 Flow agreement parameter 1 0 -1 Inline Staggered C B Two Inline Three Inline Three Staggered Time, t/T 0 .2 1 .4 .6 .8 Fluid velocity (y-component) Two Inline Three Inline Three Staggered T-averaged fluid velocity .75 0 -.75 tailbeat velocity of inphase swimmer 0 2 -2 -1 1 Time, t/T 0 .2 1 .4 .6 .8 Time, t/T 0 .2 1 .4 .6 .8 Figure 8.1: A. Flow agreement parameter parameter for three and four inline swimmers. For three swimmers, swimmer 3 stabilizes at the location with the highest flow agreement. For four inline and inphase swimmers, swimmer 4 cannot maintain its position in the group and separates from the rest of the formation. The decreased cohesion of the formation is predicted by the weak flow agreement parameter behind three swimmers. When placed in the staggered formation, swimmer 4 stabilizes close to highest flow agreement behind the 3 leading swimmers. B. Period average x-component and y-component fluid velocity in the wake of 2 / 3 inline swimmers and 3 staggered swimmers. C. Time evolution of y-component of the flow field and flapping speed at the predicted first equilibrium in the wake of 2 / 3 inline swimmers and 3 staggered swimmers. 1 3 2 4 collision 1 3 2 4 1 3 2 4 1 3 2 4 Alternative passive formations 4.I 4.II 4.III 4.IV Figure 8.2: Alternative variations of the inline formation. By introducing a small lateral offset of ℓ = 0.25 between different pairs and arranging them in a side by side formation, the trailing swimmer no longer separates from the group. In the first formation 4.I, there is collision between swimmer 3 and 4. 70 four swimmers maintain cohesive formation, but at increased power expenditure to swimmer 4. Alterna- tively, swimmers 2 and 3 or swimmers 3 and 4 can pair up to create passive formations with hydrodynamic benefits to all trailing swimmers. Stability via active phase control Next, inspired by the VPM strategy [78], we test whether active phase control is a viable strategy for swimmer 4 to maintain cohesion and generate power savings in a group of four swimmers. To this end, we formulated a feedback control law where the swimmer adjusts its phase of flapping in response to local flow to maximize the local value of V. Consider a flapping swimmer whose vertical speed ˙ y depends on time and its flapping phase ϕ . Con- sider that the swimmer can ‘sense’ or measure the agreement of its flapping motion with the fluid velocity u generated at its location (say at its midpoint) by neighboring or upstream swimmers, over a time span of m flapping periods. The goal of the swimmer would be to adjust its current flapping phase ϕ to a desired phaseΦ that maximizes its agreement with the local velocity Φ( t)= argmax ϕ 1 mT R t min(t− mT,0) v(t ′ ,ϕ )· u(t ′ ) dt ′ 1 mT R t min(t− mT,0) v(t ′ ,ϕ )· v(t ′ ,ϕ ) dt ′ ., (8.1) using a proportional phase controller adapted from [77, 78], ¨ ϕ (t)=− γ 2 [ϕ (t)− Φ( t)]− 2γ ˙ ϕ (t); (8.2) Here, Φ( t) is the desired phase andγ is a constant that determines the speed of convergence. We chose the parameters as follows: the number of periods m that describes the memory of the swimmer of the ambient fluid u is chosen to be two, such that the time history is two times the pitching period2T . The control gainγ = 3 is chosen to ensure that the actual phaseϕ (t) can reach the desired phaseΦ( t) at1% of relative error within1.5T . 71 Phase control separation 1 3 2 4 Inline 1 3 2 4 φ = 0 φ = φ * 0 1 2 3 Spacing, d /UT 1-2 2-3 3-4 Spacing, d /UT 0 1 2 3 1-2 2-3 3-4 phase desired phase Phase 2̟ 0 ̟ 0 10 20 30 40 50 Time, t/T 0 10 20 30 40 50 Time, t/T Phase control Inline A C B D 0 10 20 30 40 50 Time, t/T Figure 8.3: Flow agreement parameter field and snapshot of the swimmers and the free vortex sheets in group of A. four passive inline swimmers andB. four inline swimmers with the trailing swimmer under the proposed phase controller. Time evolution of pairwise spacing in theC. passive swimmers andD. controlled case. When applied to swimmer 4 in a group of four swimmers, this phase controller leads to a coherent stable formation (Fig .8.3), but at a higher power expenditure to swimmer 4, in fact, at 100% more power compared to solitary swimming, whereas the power savings of swimmers 2 and 3 remained robustly at 65% and 122.4%, as in the formation without swimmer 4. These observations can be explained by the fact that, when a trailing swimmer flaps at a different phase, the regions of positive and negative flow agreement parameters shift in the streamwise direction, but are not altered substantially, because changing the phase provides no mechanism for modifying the oncoming flow field created by upstream swimmers. 8.2 Cooperativeandselfishenergysavings Next, we systematically explore the emergent dynamics of inline, staggered, and phalanx patterns, by gradually increasing the number of swimmers and solving the FSI with all-to-all coupling among swim- mers. The time-evolution of pairwise distances and the flow agreement parameter for inline, staggered 72 and phalanx formations, are shown in Fig. 8.4, 8.5, and 8.6, respectively. To calculate the flow agreement parameter at the location of a swimmerj from simulations with all-to-all coupling, we subtract the self- generated flow u self from the flow field u generated by all swimmers and calculateV based on the flow velocity u− u self at the location of thej th swimmer. The power saving and flow agreement parameters values in the steady state are reported in Fig. 8.7. There’s a clear correlation between power saving and the flow agreement parameter for each swimmer in the formation. Further, for inline formations, the splitting of swimmer 4 from the formation is reflected in that the value of the flow agreement parameter of swimmer 4 goes to zero. In a group of five, the last two swimmers split and form their own subgroup, and in a group of six, the last three split and form a separate subgroup (Fig. 8.4). In all examples, swimmer 4 consistently loses hydrodynamic advantage and serves as local leader of the trailing subgroup. Considering the same cases, but with swimmers staggered at alternating lateral offset ℓ = 0.25L (Fig. 8.5), allows swimmer 4 to maintain its position in formation, albeit at no hydrodynamic benefits and slightly larger separation distance from the swimmer ahead compared to other members of the group. This spatial staggering works consistently in groups of 5 and 6. When starting side-by-side, the swimmers reach stable steady-state formations reminiscent of the pha- lanx configurations observed experimentally by Ashraf et al. in flowing water [4]. The formation main- tains its stability and cohesivness as the number of swimmers increases (Fig. 8.6). The swimmers in this configuration save power compared to solitary swimming (Fig. 8.7C). Middle swimmers gain the most hy- drodynamic advantage (up to 55% power saving for the middle swimmers in a school of six) while the two edge swimmers benefit the least. The overall trend of power saving among group members is robust to the total number of swimmers in the group. The power saving in the inline, staggered and Phalanx formations as a function of the school size is shown in Fig. 8.7C and Table 8.1. —- 73 Figure 8.4: Pairwise spacing and flow agreement parameter experienced by followers in inline formations with 3, 4, 5 and 6 swimmers, from top to bottom, respectively. A.,B. For 3 swimmers, the formation is stable with every pair stabilizing at distances d/UT =1. The flow agreements parameters are positive for second and third swimmers. C.,D. For 4 swimmers, the group splits into a group of 3 swimmers and the fourth swimmer separates from the rest. The flow agreement parameter for the fourth swimmer converges to zero. E., F. For 5 swimmers, the group splits into two groups of 3 and 2 swimmers. The flow agreement parameter is positive for all swimmers except for the fourth one. E.,F. For 5 swimmers, the group splits into two groups of 3 and 2 swimmers. The flow agreement parameter is positive for all swimmers except for the fourth one. G., H. For 6 swimmers, the group splits into two groups of 3 swimmers. The flow agreement parameter is positive for all swimmers except for the fourth one. In steady state the the two groups dynamics are similar and independent. To compute the flow agreement parameter for every swimmer we consider the flow generated by all the other swimmers and we normalize it by the number of other swimmers in the groupN− 1. From top to bottom (3 to 6 swimmers), the average speed change of the group relative to a single swimmer are− 1.59%,− 2.02%,− 2.29%,− 3.86% and the average power savings are59.77%,43.65%,47.19%,53.82%, respectively. 74 Figure 8.5: Pairwise spacing and flow agreement parameter experienced by followers in staggered formations with 3, 4, 5 and 6 swimmers, from top to bottom, respectively. A., B. For 3 swimmers, the formation is stable with every pair stabilizing at distancesd/UT = 1. The flow agreements parameters are positive for second and third swimmers. Unlike the inline formation results in Fig.??, the staggered formations with more than 3 swimmers are stable and do not split. All the swimmers experience positive non-zero flow agreement parameters. C., D. For 4 swimmers, E., F. 5 swimmers, and G., H. 6 swimmers the group remains cohesive. As the number of swimmers increase the oscillations of the trailing swimmers are more amplified. In all of the cases the third swimmer experiences the highest flow agreement parameter, while the fourth swimmer has the lowest flow agreement parameter. To compute the flow agreement parameter for every swimmer we consider the flow generated by all the other swimmers and we normalize it by the number of other swimmers in the groupN− 1. From top to bottom (3 to 6 swimmers), the average speed change of the group relative to a single swimmer are− 1.95%,− 2.53%,− 2.99%,− 3.02% and the average power savings are61.41%,41.33%,40.44%,28.48%, respectively. 75 Figure 8.6: Pairwise spacing and flow agreement parameter experienced by swimmers in Phalanx formations with 3, 4, 5 and 6 swimmers, from top to bottom, respectively. In all of the groups the formations are stable and the distances between every pair is very close to zero (side by side). The flow agreement parameter is lowest for the two marginal swimmers (around 0.25) and highest for all the other swimmers that are not at the boundaries (around 0.75). All the swimmers experience positive flow agreements. From top to bottom (3 to 6 swimmers), the average speed change of the group relative to a single swimmer are +1.03%,+2.17%,+2.95%,+3.49% and the average power savings are28.07%,33.47%,37.2%,39.72%, respectively. Table 8.1: Average power saving of groups of swimmers shown in Figs. 8.4, 8.5, 8.6 relative to a solitary swimmer Schoolsize Inline Staggered Phalanx 2 31.74% 34.72% 16.31% 3 59.77% 61.41% 28.07% 4 43.65% 41.33% 33.47% 5 47.19% 40.44% 37.2% 6 53.82% 28.48% 39.72% 76 Flow agreement Inline Staggered Phalanx Power saving 12 123 1234 12345 123456 1 2 3 4 5 6 1 3 2 4 5 6 1 3 2 4 5 6 B Staggered Inline Phalanx 0 0.8 1.6 0% 50% 100% 12 123 1234 12345 123456 12 123 1234 12345 123456 A C Figure 8.7: Emergent formation in large groups.A.,B. Snapshots the inline, staggered, and phalanx formations with 6 swimmers at steady state. C. Power saving and flow agreement parameter of each individual swimmer in inline, staggered, and phalanx formations with number of inphase swimmers ranging from 2 to 6. 8.3 Theeffectofflowparametersonemergentformations Lastly, we investigate the effect of dissipation time on the size of cohesive groups of swimmers. First, we look at how the dissipation time affects a solitary and pair of swimmers. In Fig. 8.8 and 8.9, we show a snapshot of the flow field and the flow agreement parameter field behind a solitary and a pair of free swim- mers, for dissipation times T diss = 2.45,3.45,4.45,9.45. As discussed in Chapter 6, a larger dissipation time implies higher Reynolds number and therefore a longer lasting wake This is consistent with Fig. 8.8 and 8.9. Longer lasting wakes create a flow field that travels further downstream, but this does not alter the swimming dynamics of a solitary swimmer or self-organization in pair of swimmers. As shown in Fig. 8.9, for all values of dissipation time, the pair stabilize around the same equilibrium distance of d/UT ≈ 1. Furthermore, in Tables 8.2, we show steady state values of swimming speed and hydrodynamic power for a solitary swimmer and separation distance and average power saving for a pair of swimmers with increasing dissipation time values. These quantities are very robust to dissipation time with maximum variations always less than5%. Next, we probe the role of dissipation time on the largest cohesive group size for inline swimmers. Namely, we increase the dissipation time from T diss = 2.45 to T diss = 4.45 and uniformly increase the 77 0.01 −0.01 0 VS circulation ×(U/L) Flow agreement parameter 1 0 −1 A T diss = 9.45 0 2 4 6 5 3 1 Separation distance, d /UT 0 2 4 6 5 3 1 Separation distance, d /UT B C D T diss = 4.45 T diss = 3.45 T diss = 2.45 Figure 8.8: Snapshots of flow field (left column) and flow agreement parameter V (right column) in the wake of a solitary swimmer in the VS model, for different dissipation times T diss =2.45 (A.),3.45 (B.),4.45 (C.), and9.45 (D.). 78 0.01 −0.01 0 VS circulation ×(U/L) Flow agreement parameter 1 0 −1 A 0 2 4 6 5 3 1 Separation distance, d /UT 0 2 4 6 5 3 1 Separation distance, d /UT B C D T diss = 2.45 T diss = 3.45 T diss = 4.45 T diss = 9.45 Figure 8.9: Snapshots of flow field (left column) and flow agreement parameter V (right column) in the wake of a pair of swimmers in the VS model, for different dissipation times T diss =2.45 (A.),3.45 (B.),4.45 (C.), and9.45 (D.). 79 Table 8.2: Swimming speed and hydrodynamic power of a solitary swimmer and equilibrium separation distance and average power saving in a pair of swimmers for different dissipation times T diss . T diss swimming speed hydrodynamic power 2.45 2.32 1.85 3.45 2.31 1.85 4.45 2.31 1.85 9.45 2.32 1.82 ∞ 2.47 1.89 T diss d/UT average power saving 2.45 0.99 31.78 % 3.45 1.00 30.2 % 4.45 1.00 28.98% 9.45 1.00 31.37% ∞ 1.04 29.72 % Figure 8.10: Maximum number of swimmers in a cohesive group of inline swimmers is dependent on the dissipation time. A-C. For dissipation timesT diss =2.45,3.45 and4.45, the 4th, 5th and 6th swimmers separate from the group, respectively. number of inline swimmers until the most downstream one separates from the rest of the group (see Fig. 8.10. As expected, by increasing the dissipation time and hence the downstream distance the wake can travel, the stability of the downstream swimmer increases. The size of largest cohesive group of inline swimmers is 3, 4 and 5 forT diss =2.45,3.45,4.45, respectively. This study helps us identify schooling properties that are scalable independently of the flow conditions (i.e. viscosity) and detect dynamics that depend on the fluid medium. 80 8.4 Discussion We explored the stability and energetic benefits associated with large schools where the number of swim- mers is great than two. We found that for our choice of dissipation time, inline groups with more than three swimmers are unstable and the trailing swimmer will separate from the rest. However, we found that this can be remedied by placing the swimmers in a staggered formations commonly observed in groups of flying birds [109]. Furthermore, by looking at the wake produced by three inline swimmers and the corresponding flow agreement parameter, we proposed a hydrodynamic explanations for why the trailing swimmer fails to maintain its position in the school. Next, we systematically increased the number of swimmers in the inline, staggered and Phalanx formations from 1 to 6 swimmers and computed the flow agreement parameter and power saving for each swimmer. We found that compared to the phalanx for- mation, members of the inline groups (here the third member) save significantly more energy, with some members getting a ‘free ride,’ but power savings are more ’fairly’ distributed in the phalanx formation, except for the two edge swimmers. Importantly, inline formations are not robust to group size, whereas the phalanx configuration are. 81 Chapter9 Conclusionandoutlook We investigated how mechanical and flow interaction facilitate collective locomotion in a physics-based approach. The findings and contributions of this thesis can be summarized as follows. In part one, inspired by the neuromenchanics of sea stars, we studied the collective transport of a rigid body by arrays of soft actuators. We developed a mathematical model of each tube foot as a soft actuator, consisting of active and passive force elements, that can actively extend or contract, generating active pulling or pushing forces on the substrate and the sea star body. We then studied the dynamics of the sea star driven by an array of such soft actuators. Tube feet were controlled according to our designed hierarchical motor control mode, where the direction of motion is globally-communicated to all the feet, while each foot is actuated according to local sensory motor feedback loops. The mechanosensory information that tube feet use in the local sensory motor feedback loops were minimal and local, meaning that there is no communication of sensory information between the feet. The only coupling between the tube feet comes from their mechanical connection to the same body that they are attached to. We found that the collective effect of the tube feet can lead to stable crawling motion of the sea star body. The model also exhibited robustness to perturbations in initial condition and heterogeneity in the ability of the tube feet to generate active forces, as well as to irregularities in the substrate geometry. Furthermore, we found that by following this hierarchical motor control, tube feet can transition to a different locomotion gait 82 called the bouncing gait. This gait which is observed across different species of sea stars, is characterized by clear vertical bounces, increased locomotion speed and high coordination in the actuation of tube feet. All of these characteristics were shown in our model as well. Next, we focused on finding a set of mechanosensory cues that are most effective for local sensory- motor feedback control of the tube feet. Namely, we used a model-free Reinforcement Learning (RL) approach where we trained the tube feet optimal control policies using a set of candidate sensory cues. We found that axial and shear strain are the most effective cues for the tube feet to learn when to detach and transition to recovery stroke and when to push or pull during the power stroke. Training an optimal control policy using these mechanosensory cues, we arrived at a control policy that can be generalized to hundreds of tube feet without any modification to the model. We also showed that the model is robust to the introduction of a random recovery stroke period in the tube feet. Finally, we investigated tube feet as adaptive force generators capable of adjusting their active push/pull force ratio in order to walk on vertical or even inverted substrates. In the second half of this thesis, we turned to the question of coordination in schools of fish. Specifi- cally, we examined the pairwise interactions of flapping rigid swimmers in the context of a vortex sheet model. We observed that freely swimming plates, without any active feedback control, always reach equi- librium separation distances, regardless of their phase lag or lateral offset. In these emergent equilibria, the phase and the spacing between the pair always follows a linear relationship. This linear phase-spacing relationship is a universal behavior reported in several experimental and numerical studies. Furthermore, we found that the swimmers always save power and form stable and cohesive pairs. Next, we found that the wake of a solitary swimmer contains valuable information about destructive and constructive inter- action between the follower swimmer and the wake of the leader. This information can be decoded to provides accurate predictions of the pairwise formations, without the need to solve the complicated two- way coupled fluid-structure interactions. Finally, we explored the emergent formations in large groups 83 of swimmers, in the inline, staggered and phalanx formations. Our findings show that in the inline or staggered formation, swimmers have an uneven distribution of benefits with some swimmers using more power than a solitary one and some getting a free ride with zero cost. On the other hand, in the phalanx formation, power savings are more fairly distributed among the group members. Our findings emphasize the role of mechanical and flow interactions in the passive emergence of coop- erative transport and collective locomotion. Our mathematical models not only provide a deeper insight into collective behavior in animals groups, but also have potential applications to the design and develop- ment of multi-agent robotic systems with decentralized control. 84 References [1] David E Alexander. Nature’s Machines: An Introduction to Organismal Biomechanics. Academic Press, 2017. [2] Yasmin S Appelhans, Jörn Thomsen, Christian Pansch, Frank Melzner, and Martin Wahl. “Sour times: seawater acidification effects on growth, feeding behaviour and acid–base status of Asterias rubens and Carcinus maenas”. In: Marine Ecology Progress Series 459 (2012), pp. 85–98. doi: 10.3354/meps09697. [3] G Arranz, O Flores, and M Garcıa-Villalba. “Flow interaction of three-dimensional self-propelled flexible plates in tandem”. In: Journal of Fluid Mechanics 931 (2022). [4] Intesaaf Ashraf, Hanaé Bradshaw, Thanh Tung Ha, José Halloy, Ramiro Godoy-Diana, and Benjamin Thiria. “Simple phalanx pattern leads to energy saving in cohesive fish schooling”. In: Proc. Nat. Acad. Sci. 114 (2017), pp. 9599–9604.issn: 10916490.doi: 10.1073/pnas.1706503114. [5] Henry C. Astley. “Getting around when youâre round: quantitative analysis of the locomotion of the blunt-spined brittle star, Ophiocoma echinata”. In: Journal of Experimental Biology 215 (2012), pp. 1923–1929.issn: 0022-0949.doi: 10.1242/jeb.068460. [6] Karl Johan Åström. “Optimal control of Markov processes with incomplete state information”. In: Journal of Mathematical Analysis and Applications 10.1 (1965), pp. 174–205. [7] Amir Ayali, Anke Borgmann, Ansgar Bueschges, Einat Couzin-Fuchs, Silvia Daun-Gruhn, and Philip Holmes. “The comparative investigation of the stick insect and cockroach models in the study of insect locomotion”. In: Current Opinion in Insect Science 12 (2015), pp. 1–10. [8] Richard Stephen Kent Barnes, Peter P Calow, Peter JW Olive, David W Golding, and John I Spicer. The invertebrates: a synthesis. John Wiley & Sons, 2009. [9] A. D. Becker, H. Masoud, J. W. Newbolt, M. J. Shelley, and L. Ristroph. “Hydrodynamic schooling of flapping swimmers”. In: Nature Communications 6 (2015), p. 8514.issn: 2041-1723.doi: 10.1038/ncomms9514. 85 [10] Michael A. Bell, Isabella Pestovski, William Scott, Kitty Kumar, Mohammad K. Jawed, Derek A. Paley, Carmel Majidi, James C. Weaver, and Robert J. Wood. “Echinoderm-inspired tube feet for robust robot locomotion and adhesion”. In: IEEE Robotics and Automation Letters 3 (2018), pp. 2222–2228.issn: 2377-3766.doi: 10.1109/lra.2018.2810949. [11] Amneet Pal Singh Bhalla, Rahul Bale, Boyce E Griffith, and Neelesh A Patankar. “A unified mathematical framework and an adaptive numerical method for fluid–structure interaction with rigid, deforming, and elastic bodies”. In: Journal of Computational Physics 250 (2013), pp. 446–476. [12] Amneet Pal Singh Bhalla, Nishant Nangia, Panagiotis Dafnakis, Giovanni Bracco, and Giuliana Mattiazzo. “Simulating water-entry/exit problems using Eulerian–Lagrangian and fully-Eulerian fictitious domain methods within the open-source IBAMR library”. In: Applied Ocean Research 94 (2020), p. 101932. [13] John Binyon. “Chapter 8 - Physiology of the water vascular system and the neural control of locomotion”. In: Physiology of Echinoderms. International Series of Monographs in Pure and Applied Biology. Division: Zoology. Pergamon, 1972, pp. 124–146.doi: https://doi.org/10.1016/B978-0-08-016991-0.50015-5. [14] Robert William Blake. “Fish functional design and swimming performance”. In: Journal of fish biology 65.5 (2004), pp. 1193–1222. [15] Theodore Bullock and G Adrian Horridge. “Structure and function in the nervous systems of invertebrates.” In: (1965). [16] Marcello Calisti, Andrea Arienti, Federico Renda, Guy Levy, Binyamin Hochner, Barbara Mazzolai, Paolo Dario, and Cecilia Laschi. “Design and development of a soft robot with crawling and grasping capabilities”. In: 2012 IEEE International Conference on Robotics and Automation (2012), pp. 4950–4955.doi: 10.1109/icra.2012.6224671. [17] Ching-Ping Chou and B. Hannaford. “Measurement and modeling of McKibben pneumatic artificial muscles”. In: IEEE Transactions on Robotics and Automation 12 (1996), pp. 90–102.issn: 1042-296X.doi: 10.1109/70.481753. [18] Elizabeth G. Clark, Daichi Kanauchi, Takeshi Kano, Hitoshi Aonuma, Derek E. G. Briggs, and Akio Ishiguro. “The function of the ophiuroid nerve ring: how a decentralized nervous system controls coordinated locomotion”. In: Journal of Experimental Biology 222 (2018).issn: 0022-0949. doi: 10.1242/jeb.192104. [19] JLS Cobb. “Neurobiology of the Echinodermata”. In: Nervous systems in invertebrates. Springer, 1987, pp. 483–525. [20] J. Edward Colgate and Kevin M. Lynch. “Mechanics and control of swimming: A review”. In: IEEE Journal of Oceanic Engineering 29.3 (2004), pp. 660–673.issn: 0364-9059.doi: 10.1109/joe.2004.833208. [21] Steve Collins, Andy Ruina, Russ Tedrake, and Martijn Wisse. “Efficient bipedal robots based on passive-dynamic walkers”. In: Science 307 (2005), pp. 1082–1085.issn: 0036-8075.doi: 10.1126/science.1107799. 86 [22] Steven H. Collins, Martijn Wisse, and Andy Ruina. “A three-dimensional passive-dynamic walking robot with two legs and knees”. In: The International Journal of Robotics Research 20 (2001), pp. 607–615.issn: 0278-3649.doi: 10.1177/02783640122067561. [23] Brendan Colvert and Eva Kanso. “Fishlike rheotaxis”. In: Journal of Fluid Mechanics 793 (2016), pp. 656–666.doi: 10.1017/jfm.2016.141. [24] Longzhen Dai, Guowei He, Xiang Zhang, and Xing Zhang. “Stable formations of self-propelled fish-like swimmers induced by hydrodynamic interactions”. In: Journal of The Royal Society Interface 15.147 (2018), p. 20180490. [25] Silvia Daun-Gruhn and Ansgar BÃŒschges. “From neuron to behavior: dynamic equation-based prediction of biological processes in motor control”. In: Biological Cybernetics 105 (2011), p. 71. issn: 0340-1200.doi: 10.1007/s00422-011-0446-6. [26] O Ellers, AS Johnson, and T Motokawa. “Do general theories of locomotion apply to underwater walkers?” In: Integrative and Comparative Biology 58 (2018), E57–E57. [27] O. Ellers, S. Heydari, K. Ellers, A. Johnson, E. Kanso, and M.J. McHenry. “Soft skeletons transmit force with variable gearing.” In: (). [28] O. Ellers, AS. Johnson, K. Guttenplan, and T. Motokawa. “The bounce in a seastars step: Classifying gaits in underwater legged locomotion”. In: Integrative and Comparative Biology 54 (2014), E59–E59.doi: 10.1093/icb/icu008. [29] R Etzel, J Khoriaty, O Ellers, and AS Johnson. “The contribution of morphological characteristics on the bouncing gait of sea stars: A cross-species comparison”. In: Society of Integrative and Comparative Biology 59 (2019), E114–E3114. [30] F. Fang. “Hydrodynamic interactions between self-propelled flapping wings”. PhD thesis. New York University, 2016. [31] Daniel Floryan, Tyler Van Buren, Clarence W Rowley, and Alexander J Smits. “Scaling the propulsive performance of heaving and pitching foils”. In: Journal of Fluid Mechanics 822 (2017), pp. 386–397. [32] Yuan-Cheng Fung and Richard Skalak. “Biomechanics. Mechanical Properties of Living Tissues”. In: Journal of Applied Mechanics 49 (1982), pp. 464–465.issn: 0021-8936.doi: 10.1115/1.3162171. [33] Nathaniel N. Goldberg, Xiaonan Huang, Carmel Majidi, Alyssa Novelia, Oliver M. O’Reilly, Derek A. Paley, and William L. Scott. “On planar discrete elastic rod models for the locomotion of soft robots”. In: Soft Robotics (2019).issn: 2169-5172.doi: 10.1089/soro.2018.0104. [34] Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. “Variance reduction techniques for gradient estimates in reinforcement learning”. In: Journal of Machine Learning Research 5.Nov (2004), pp. 1471–1530. 87 [35] Boyce E Griffith, Richard D Hornung, David M McQueen, and Charles S Peskin. “An adaptive, formally second order accurate version of the immersed boundary method”. In: Journal of computational physics 223.1 (2007), pp. 10–49. [36] Boyce E Griffith and Neelesh A Patankar. “Immersed methods for fluid–structure interaction”. In: Annual review of fluid mechanics 52 (2020), p. 421. [37] Boyce E Griffith and Charles S Peskin. “On the order of accuracy of the immersed boundary method: Higher order convergence rates for sufficiently smooth problems”. In: Journal of Computational Physics 208.1 (2005), pp. 75–105. [38] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”. In: Proceedings of the 35th International Conference on Machine Learning. Ed. by Jennifer Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Research. StockholmsmÀssan, Stockholm Sweden: PMLR, Oct. 2018, pp. 1861–1870.url: http://proceedings.mlr.press/v80/haarnoja18b.html. [39] Elise Hennebert, Romana Santos, and Patrick Flammang. “Echinoderms don’t suck: evidence against the involvement of suction in tube foot attachment”. In:Zoosymposia 7.1 (2012), pp. 25–32. [40] James E Herbert-Read, Andrea Perna, Richard P Mann, Timothy M Schaerf, David JT Sumpter, and Ashley JW Ward. “Inferring the rules of interaction of shoaling fish”. In: Proceedings of the National Academy of Sciences 108.46 (2011), pp. 18726–18731. [41] Sina Heydari and Eva Kanso. “School cohesion, speed and efficiency are modulated by the swimmers flapping motion”. In: Journal of Fluid Mechanics 922 (2021), A27.doi: 10.1017/jfm.2021.551. [42] Archibald Vivian Hill. “The heat of shortening and the dynamic constants of muscle”. In: Proceedings of the Royal Society of London. Series B - Biological Sciences 126 (1938), pp. 136–195. issn: 2053-9193.doi: 10.1098/rspb.1938.0050. [43] Alexander P Hoover, Ricardo Cortez, Eric D Tytell, and Lisa J Fauci. “Swimming performance, resonance and shape evolution in heaving flexible panels”. In: Journal of Fluid Mechanics 847 (2018), pp. 386–416. [44] Alexander P Hoover, Joost Daniels, Janna C Nawroth, and Kakani Katija. “A Computational Model for Tail Undulation and Fluid Transport in the Giant Larvacean”. In: Fluids 6.2 (2021), p. 88. [45] Alexander P Hoover and Eric Tytell. “Decoding the relationships between body shape, tail beat frequency, and stability for swimming fish”. In: Fluids 5.4 (2020), p. 215. [46] Y. Huang, M. Nitsche, and E. Kanso. “Hovering in oscillatory flows”. In: JournalofFluidMechanics 804 (2016), pp. 531–549. [47] Y. Huang, L. Ristroph, M. Luhar, and E. Kanso. “Bistability in the rotational motion of rigid and flexible flyers”. In: Journal of Fluid Mechanics 849 (2018), pp. 1043–1067. 88 [48] IBAMR. IBAMR:An adaptive and distributed-memory parallel implementation of the immersed boundary (IB) method. https://ibamr.github.io. [49] Fumiya Iida, Rolf Pfeifer, and André Seyfarth. “AI in locomotion: Challenges and perspectives of underactuated robots”. In: 50 years of artificial intelligence . Springer, 2007, pp. 134–143. [50] Fumiya Iida, Rolf Pfeifer, Luc Steels, and Yasuo Kuniyoshi. Embodied artificial intelligence: International seminar, Dagstuhl Castle, Germany, July 7-11, 2003, Revised Selected Papers. Vol. 3139. Springer, 2004. [51] Tommi Jaakkola, Satinder P Singh, and Michael I Jordan. “Reinforcement learning algorithm for partially observable Markov decision problems”. In: Advances in neural information processing systems. 1995, pp. 345–352. [52] Yusheng Jiao, Feng Ling, Sina Heydari, Nicolas Heess, Josh Merel, and Eva Kanso. “Learning to swim in potential flow”. In: Physical Review Fluids 6.5 (2021), p. 050505. [53] AS Johnson, O Ellers, R Etzel, and J Khoriaty. “The oscillatory gait of high-speed sea stars: Do sea stars of varying morphology vary stride length or step frequency to change speed?” In: Society of Integrative and Comparative Biology 59 (2019), E310–E310. [54] M. A. Jones. “The separated flow of an inviscid fluid around a moving flat plate”. In: Journal of Fluid Mechanics 496 (2003), p. 405. [55] M. J. Jones M. A .and Shelley. “Falling cards”. In: Journal of Fluid Mechanics 540 (2005), pp. 393–425. [56] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. “Planning and acting in partially observable stochastic domains”. In: Artificial intelligence 101.1-2 (1998), pp. 99–134. [57] Sham Kakade. “A Natural Policy Gradient”. In: Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic. NIPS’01. Vancouver, British Columbia, Canada: MIT Press, 2001, 1531â1538. [58] Takeshi Kano, Daichi Kanauchi, Hitoshi Aonuma, Elizabeth G. Clark, and Akio Ishiguro. “Decentralized control mechanism for determination of moving direction in brittle Stars with penta-radially symmetric body”. In: Frontiers in Neurorobotics 13 (2019), p. 66.doi: 10.3389/fnbot.2019.00066. [59] Takeshi Kano, Ryo Yoshizawa, and Akio Ishiguro. “Tegotae-based decentralised control scheme for autonomous gait transition of snake-like robots”. In: Bioinspiration & Biomimetics 12 (2017), p. 046009.issn: 1748-3190.doi: 10.1088/1748-3190/aa7725. [60] E Kanso and A C H Tsang. “Pursuit and Synchronization in Hydrodynamic Dipoles”. In: Journal of Nonlinear Science 25(5) (2015), p. 1141. [61] Y. Katz, K. Tunstrøm, C. C. Ioannou, C. Huepe, and I. D. Couzin. “Inferring the structure and dynamics of interactions in schooling fish”. In: Proceedings of the National Academy of Sciences 108.46 (2011), pp. 18720–18725. 89 [62] G.A. Kerkut. “The mechanisms of coordination of the starfish tube feet”. In: Behaviour 6 (1954), pp. 206–232.issn: 0005-7959.doi: 10.1163/156853954x00103. [63] G.A. Kerkut. “The retraction and protraction of the tube feet of the starfish, ( Asterias Rubens L.)” In: Behaviour 8 (1955), pp. 112–129.issn: 0005-7959.doi: 10.1163/156853955x00193. [64] GA Kerkut. “The forces exerted by the tube feet of the starfish during locomotion”. In: Journal of Experimental Biology 30.4 (1953), pp. 575–583. [65] William M Kier. “Hydrostatic skeletons and muscular hydrostats”. In: nautilus 8.11 (1992), p. 205. [66] William M. Kier and Kathleen K. Smith. “Tongues, tentacles and trunks: the biomechanics of movement in muscularâhydrostats”. In: Zoological Journal of the Linnean Society 83 (1985), pp. 307–324.issn: 1096-3642.doi: 10.1111/j.1096-3642.1985.tb01178.x. [67] Sohae Kim, Wei-Xi Huang, and Hyung Jin Sung. “Constructive and destructive interaction modes between two tandem flexible flags in viscous flow”. In: Journal of fluid mechanics 661 (2010), pp. 511–521. [68] Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization”. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015.url: http://arxiv.org/abs/1412.6980. [69] Vijay Konda and John Tsitsiklis. “Actor-Critic Algorithms”. In: Advances in Neural Information Processing Systems. Ed. by S. Solla, T. Leen, and K. Müller. Vol. 12. Cambridge, MA, USA: MIT Press, 2000.url: https://proceedings.neurips.cc/paper/1999/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf. [70] Melike Kurt, Amin Mivehchi, and Keith W Moored. “Two-dimensionally stable self-organization arises in simple schooling swimmers through hydrodynamic interactions”. In: arXiv preprint arXiv:2102.03571 (2021). [71] Cecilia Laschi, Matteo Cianchetti, Barbara Mazzolai, Laura Margheri, Maurizio Follador, and Paolo Dario. “Soft robot arm inspired by the octopus”. In: Advanced Robotics 26 (2012), pp. 709–727.issn: 0169-1864.doi: 10.1163/156855312x626343. [72] George V. Lauder. “Fish Locomotion: Recent Advances and New Directions”. In: Annual Review of Marine Science 7.1 (2015), pp. 1–25.issn: 1941-1405.doi: 10.1146/annurev-marine-010814-015614. [73] Jae H Lee, Alex D Rygg, Ebrahim M Kolahdouz, Simone Rossi, Stephen M Retta, Nandini Duraiswamy, Lawrence N Scotten, Brent A Craven, and Boyce E Griffith. “Fluid–structure interaction models of bioprosthetic heart valve dynamics in an experimental pulse duplicator”. In: Annals of biomedical engineering 48.5 (2020), pp. 1475–1490. [74] Birgit Lengerer, Morgane Algrain, Mathilde Lefevre, JérÃŽme Delroisse, Elise Hennebert, and Patrick Flammang. “Interspecies comparison of sea star adhesive proteins”. In: Philosophical Transactions of the Royal Society B: Biological Sciences 374 (2019), p. 20190195.issn: 0962-8436. doi: 10.1098/rstb.2019.0195. 90 [75] Guy Levy, Tamar Flash, and Binyamin Hochner. “Arm coordination in octopus crawling involves unique motor control strategies”. In: Current Biology 25 (2015), pp. 1195–1200.issn: 0960-9822. doi: 10.1016/j.cub.2015.02.064. [76] Guy Levy, Nir Nesher, Letizia Zullo, and Binyamin Hochner. “Motor control in soft-bodied animals”. In: The Oxford Handbook of Invertebrate Neurobiology. Oxford University Press, 2017. [77] Liang Li, Danshi Liu, Jian Deng, Matthew J Lutz, and Guangming Xie. “Fish can save energy via proprioceptive sensing”. In: Bioinspiration & biomimetics 16.5 (2021), p. 056013. [78] Liang Li, Máté Nagy, Jacob M. Graving, Joseph Bak-Coleman, Guangming Xie, and Iain D. Couzin. “Vortex phase matching as a strategy for schooling in robots and in fish”. In: Nature Communications 11.1 (2020), p. 5408.doi: 10.1038/s41467-020-19086-0. [79] James C Liao. “Fish swimming efficiency”. In: Current Biology 32.12 (2022), R666–R671. [80] Michael James Lighthill. “Aquatic animal propulsion of high hydromechanical efficiency”. In: Journal of Fluid Mechanics 44.2 (1970), pp. 265–301. [81] MJ Lighthill. “Mathematical biofluiddynamics: SIAM”. In: Regional Conference Series in Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia. 1975. [82] Zhonglu Lin, Amneet Pal Singh Bhalla, Boyce Griffith, Zi Sheng, Hongquan Li, Dongfang Liang, and Yu Zhang. “How swimming style affects schooling of two fish-like wavy hydrofoils”. In: arXiv preprint arXiv:2209.01590 (2022). [83] Shixin Mao, Erbao Dong, Hu Jin, Min Xu, Shiwu Zhang, Jie Yang, and Kin Huat Low. “Gait study and pattern generation of a starfish-like soft robot with flexible rays actuated by SMAs”. In: Journal of Bionic Engineering 11 (2014), pp. 400–411.issn: 1672-6529.doi: 10.1016/s1672-6529(14)60053-6. [84] S. Marras, S. S. Killen, J. Lindström, D. J. McKenzie, J. F. Steffensen, and P. Domenici. “Fish swimming in schools save energy regardless of their spatial position”. In: Behavioral ecology and sociobiology 69.2 (2015), pp. 219–226. [85] V Mashanov, O Zueva, T Rubilar, L Epherra, and JE Garcia-Arraras. “Echinodermata”. In: Structure and evolution of invertebrate nervous systems (2016), pp. 665–688. [86] Yoshiya Matsuzaka, Eiki Sato, Takeshi Kano, Hitoshi Aonuma, and Akio Ishiguro. “Non-centralized and functionally localized nervous system of ophiuroids: evidence from topical anesthetic experiments”. In: Biology Open 6 (2017), pp. 425–438.issn: 2046-6390.doi: 10.1242/bio.019836. [87] R S McCurley and W M Kier. “The functional morphology of starfish tube feet: The role of a crossed-fiber helical array in movement.” In: The Biological bulletin 188 (1995), pp. 197–209.issn: 0006-3185.doi: 10.2307/1542085. [88] M J McHenry, A Carrillo, T Po, A McKee, B Pernet, S Heydari, and Kanso E. “The biomechanics and control of walking in juvenile sea stars (Leptasterias sp.)” In: (in prep). 91 [89] Amberle McKee, Alberto P Soto, Phoebe Chen, and Matthew J McHenry. “The sensory basis of schooling by intermittent swimming in the rummy-nose tetra (Hemigrammus rhodostomus)”. In: Proceedings of the Royal Society B 287.1937 (2020), p. 20200568. [90] Amir Mirzaeinia, F Heppner, and Mostafa Hassanalian. “An analytical study on leader and follower switching in V-shaped Canada Goose flocks for energy management purposes”. In: Swarm Intelligence 14.2 (2020), pp. 117–141. [91] Rajat Mittal, Haibo Dong, Meliha Bozkurttas, FM Najjar, Abel Vargas, and Alfred Von Loebbecke. “A versatile sharp interface immersed boundary method for incompressible flows with complex boundaries”. In: Journal of computational physics 227.10 (2008), pp. 4825–4852. [92] Werner Nachtigall. Biological mechanisms of attachment: The comparative morphology and bioengineering of organs for linkage, suction, and adhesion. Springer Science & Business Media, 2013. [93] Jennifer C Nauen and George V Lauder. “Hydrodynamics of caudal fin locomotion by chub mackerel, Scomber japonicus (Scombridae).” In: The Journal of experimental biology 205.Pt 12 (2002), pp. 1709–24.issn: 0022-0949. [94] J. W. Newbolt, J. Zhang, and L. Ristroph. “Flow interactions between uncoordinated flapping swimmers give rise to group cohesion”. In: Proceedings of the National Academy of Sciences 116 (2019), p. 201816098.issn: 0027-8424.doi: 10.1073/pnas.1816098116. [95] M. Nitsche and R. Krasny. “A numerical study of vortex ring formation at the edge of a circular tube”. In: Journal of Fluid Mechanics 276 (1994), pp. 139–161. [96] Dai Owaki, Masashi Goda, Sakiko Miyazawa, and Akio Ishiguro. “A minimal model describing hexapedal interlimb coordination: the Tegotae-based approach”. In: Frontiers in Neurorobotics 11 (2017), p. 29.doi: 10.3389/fnbot.2017.00029. [97] A. U. Oza, L. Ristroph, and M. J. Shelley. “Lattices of hydrodynamically interacting flapping swimmers”. In: Physical Review X 9.4 (2019), p. 041024. [98] Virginia L Paine. “The tube feet of starfishes as autonomous organs”. In: The American Naturalist 63 (1929), pp. 517–529.issn: 0003-0147.doi: 10.1086/280286. [99] Sung Goon Park and Hyung Jin Sung. “Hydrodynamics of flexible fins propelled in tandem, diagonal, triangular and diamond configurations”. In: JournalofFluidMechanics 840 (2018), p. 154. [100] B. L. Partridge. “The structure and function of fish schools”. In: Scientific american 246.6 (1982), pp. 114–123. [101] B. L. Partridge and T. J. Pitcher. “Evidence against a hydrodynamic function for fish schools”. In: Nature 279.5712 (1979), pp. 418–419. 92 [102] Thibaut Paschal, Michael A. Bell, Jakob Sperry, Satchel Sieniewicz, Robert J. Wood, and James C. Weaver. “Design, fabrication, and characterization of an untethered amphibious sea urchin-inspired robot”. In: IEEE Robotics and Automation Letters 4 (2019), pp. 3348–3354.doi: 10.1109/lra.2019.2926683. [103] Ze-Rui Peng, Haibo Huang, and Xi-Yun Lu. “Hydrodynamic schooling of multiple self-propelled flapping plates”. In: Journal of Fluid Mechanics 853 (2018), pp. 587–600. [104] Charles S Peskin. “Numerical analysis of blood flow in the heart”. In: Journal of computational physics 25.3 (1977), pp. 220–252. [105] Jan Peters and Stefan Schaal. “Natural actor-critic”. In: Neurocomputing 71.7-9 (2008), pp. 1180–1190. [106] Jan Peters and Stefan Schaal. “Policy gradient methods for robotics”. In: Intelligent Robots and Systems, 2006 IEEE/RSJ International Conference on. IEEE. 2006, pp. 2219–2225. [107] Rolf Pfeifer and Josh Bongard. How the body shapes the way we think: a new view of intelligence. MIT press, 2006. [108] Matteo Pirotta, Marcello Restelli, and Luca Bascetta. “Adaptive step-size for policy gradient methods”. In: Advances in Neural Information Processing Systems. 2013, pp. 1394–1402. [109] Steven J Portugal, Tatjana Y Hubel, Johannes Fritz, Stefanie Heese, Daniela Trobe, Bernhard Voelkl, Stephen Hailes, Alan M Wilson, and James R Usherwood. “Upwash exploitation and downwash avoidance by flap phasing in ibis formation flight”. In: Nature 505.7483 (2014), pp. 399–402. [110] S. Ramananarivo, F. Fang, A. Oza, J. Zhang, and L. Ristroph. “Flow interactions lead to orderly formations of flapping wings in forward flight”. In: Phys. Rev. Fluids 1 (7 Nov. 2016), p. 071201. doi: 10.1103/PhysRevFluids.1.071201. [111] Leif Ristroph, James C Liao, and Jun Zhang. “Lateral line layout correlates with the differential hydrodynamic pressure on swimming fish”. In: Physical Review Letters 114.1 (2015), p. 018102. [112] Sara Brin Rosenthal, Colin R Twomey, Andrew T Hartnett, Hai Shan Wu, and Iain D Couzin. “Revealing the hidden networks of interaction in mobile animal groups allows prediction of complex behavioral contagion”. In: Proceedings of the National Academy of Sciences 112.15 (2015), pp. 4690–4695. [113] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. “Proximal policy optimization algorithms”. In: arXiv preprint arXiv:1707.06347 (2017). [114] Jung-Hee Seo and Rajat Mittal. “Improved swimming performance in schooling fish via leading-edge vortex enhancement”. In: Bioinspiration & Biomimetics 17.6 (2022), p. 066020. [115] E. Shaw. “Schooling fishes: the school, a truly egalitarian form of organization in which all members of the group are alike in influence, offers substantial benefits to its participants”. In: American Scientist 66.2 (1978), pp. 166–175. 93 [116] J. E. Smith. “The mechanics and innervation of the starfish tube foot-ampulla system”. In: Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 232 (1946), pp. 279–310.issn: 2054-0280.doi: 10.1098/rstb.1946.0003. [117] J. E. Smith. “The role of the nervous system in some activities of starfishes”. In: Biological Reviews 20 (1945), pp. 29–43.issn: 1469-185X.doi: 10.1111/j.1469-185x.1945.tb00312.x. [118] JE Smith. “The activities of the tube feet of Asterias rubens L: I. The mechanics of movement and of posture”. In: Journal of Cell Science 3.1 (1947), pp. 1–14. [119] Matthijs TJ Spaan. “Partially observable Markov decision processes”. In: Reinforcement Learning. Springer, 2012, pp. 387–414. [120] Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. “Policy Gradient Methods for Reinforcement Learning with Function Approximation”. In: Proceedings of the 12th International Conference on Neural Information Processing Systems. NIPS’99. Denver, CO: MIT Press, 1999, 1057â1063. [121] Jon C Svendsen, Jakob Skov, Mogens Bildsoe, and John Fleng Steffensen. “Intra-school positional preference and reduced tail beat frequency in trailing positions in schooling roach under experimental conditions”. In: Journal of fish biology 62.4 (2003), pp. 834–846. [122] Russ Tedrake, Teresa w. Zhang, Ming-Fai Fong, and H. Sebastian Seung. “Actuating a simple 3D passive dynamic walker”. In: IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004 5 (2004), pp. 4656–4661.doi: 10.1109/robot.2004.1302452. [123] Bertrand Tondu. “Modelling of the McKibben artificial muscle: A review”. In: Journal of Intelligent Material Systems and Structures 23 (2012), pp. 225–253.issn: 1045-389X.doi: 10.1177/1045389x11435435. [124] Hugh Trenchard and Matjaž Perc. “Energy saving mechanisms, collective behavior and the variation range hypothesis in biological systems: a review”. In: BioSystems 147 (2016), pp. 40–66. [125] G. S. Triantafyllou and M. S. Triantafyllou. “Reverse Kármán Streets in the Wake of Flapping Foils for Application in Fish Propulsion”. In: Bluff-Body Wakes, Dynamics and Instabilities . Springer, 1993, pp. 11–14. [126] M. S. Triantafyllou, G. S. Triantafyllou, and D. K. Yue. “Hydrodynamics of fishlike swimming”. In: Annual review of fluid mechanics 32.1 (2000), pp. 33–53. [127] A. C. H. Tsang and E. Kanso. “Dipole interactions in doubly periodic domains”. In: Journal of Nonlinear Science 23.6 (2013), pp. 971–991. [128] Hsue-Shen Tsien. “Symmetrical Joukowsky airfoils in shear flow”. In: Quarterly of Applied Mathematics 1.2 (1943), pp. 130–148. [129] Eric D Tytell, Chia-Yu Hsu, and Lisa J Fauci. “The role of mechanical resonance in the neural control of swimming in fishes”. In: Zoology 117.1 (2014), pp. 48–56. 94 [130] Eric D Tytell, Megan C Leftwich, Chia-Yu Hsu, Boyce E Griffith, Avis H Cohen, Alexander J Smits, Christina Hamlet, and Lisa J Fauci. “Role of body stiffness in undulatory swimming: insights from robotic and computational models”. In: Physical Review Fluids 1.7 (2016), p. 073202. [131] T. Van Buren, D. Floryan, and A. J. Smits. “Scaling and performance of simultaneously heaving and pitching foils”. In: AIAA Journal 57.9 (2019), pp. 3666–3677. [132] Wouter G van Veen, Johan L van Leeuwen, and Florian T Muijres. “Malaria mosquitoes use leg push-off forces to control body pitch during take-off”. In: Journal of Experimental Zoology Part A: Ecological and Integrative Physiology 333.1 (2020), pp. 38–49. [133] S. Verma, G. Novati, and P. Koumoutsakos. “Efficient collective swimming by harnessing vortices through deep reinforcement learning”. In: Proceedings of the National Academy of Sciences 115.23 (2018), pp. 5849–5854. [134] Cees J Voesenek, Gen Li, Florian T Muijres, and Johan L Van Leeuwen. “Experimental–numerical method for calculating bending moments in swimming fish shows that fish larvae control undulatory swimming with simple actuation”. In: PLoS biology 18.7 (2020), e3000462. [135] Lex Weaver and Nigel Tao. “The optimal reward baseline for gradient-based reinforcement learning”. In: arXiv preprint arXiv:1301.2315 (2013). [136] D. Weihs. “Hydromechanics of fish schooling”. In: Nature 241 (1973), 241290a0.issn: 1476-4687. doi: 10.1038/241290a0. [137] Ronald J Williams. “Simple statistical gradient-following algorithms for connectionist reinforcement learning”. In: Machine learning 8.3-4 (1992), pp. 229–256.doi: 10.1007/BF00992696. [138] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. “Variance reduction for policy gradient with action-dependent factorized baselines”. In: arXiv preprint arXiv:1803.07246 (2018). [139] Dewu Yang and Jie Wu. “Hydrodynamic Interaction of Two Self-Propelled Fish Swimming in a Tandem Arrangement”. In: Fluids 7.6 (2022), p. 208. [140] Kotaro Yasui, Kazuhiko Sakai, Takeshi Kano, Dai Owaki, and Akio Ishiguro. “Decentralized control scheme for myriapod robot inspired by adaptive and resilient centipede locomotion”. In: Plos One 12 (2017), e0171421.doi: 10.1371/journal.pone.0171421. [141] Kazuya Yoshimura, Hajimu Tsurimaki, and Tatsuo Motokawa. “Memory of direction of locomotion in sea urchins: effects of nerves on direction and activity of tube feet”. In: Marine Biology 165 (2018), p. 84.issn: 0025-3162.doi: 10.1007/s00227-018-3342-y. [142] Tingting Zhao, Hirotaka Hachiya, Gang Niu, and Masashi Sugiyama. “Analysis and improvement of policy gradient estimation”. In: Advances in Neural Information Processing Systems. 2011, pp. 262–270. 95 [143] Xiaojue Zhu, Guowei He, and Xing Zhang. “Flow-Mediated Interactions between Two Self-Propelled Flapping Filaments in Tandem Configuration”. In: Physical Review Letters 113.23 (2014), p. 238105.issn: 0031-9007.doi: 10.1103/physrevlett.113.238105. [144] Olga Zueva, Maleana Khoury, Thomas Heinzeller, Daria Mashanova, and Vladimir Mashanov. “The complex simplicity of the brittle star nervous system”. In: Frontiers in Zoology 15 (2018), p. 1. issn: 1742-9994.doi: 10.1186/s12983-017-0247-4. 96 AppendixA Cooperativetransportusingtubefeetinspiredactuators A.1 ReinforcementLearningAlgorithm We consider the problem where a learning agent controls a body to move and behave in interaction with a surrounding environment. Reinforcement learning refers to the setting where the learning agent must learn from experiences for which it receives scalar reward feedback when it produces behavior. That is, experience is acquired by the agent from repeated interactions of the body with the environment, in a trial-and-error fashion, during which the agent collects observations, actions, and rewards. The goal of the agent is to learn to produce behavior that maximizes rewards. The agent is model-free when its learning and acting does not make use of eitherapriori or developed knowledge of the physics of the system. Here we give an overview of the mathematical foundations of the method we are using and provide the details for the specific algorithm used. The mathematical description of the algorithm is taken from [52]. The physical laws governing the motion of an animal or robot are described by a (potentially stochastic) transition ruleP(s t+1 |s t ,a t ) that evolves a set of statess t to new statess t+1 based on actionsa t instructed by the agent to control the animal or robot. Such a discrete time control process is known in mathematics as a Markov decision process (MDP). When the agent has access to only partial observationso t =o(s t ) of the states t , it is called a Partially Observable Markov Decision Process (POMDP) [6, 56, 51, 119] 97 Mathematical Foundation The RL problem consists of learning a policy π θ (a t |s t ) that maximizes a given objective. Here we focus on the infinite horizon objective based on the sum of discounted future rewards, R t = ∞ X t ′ =t γ t ′ − t r t ′, (A.1) whereR t is referred to as thereturn. The parameterγ ∈[0,1) is known as thediscountfactor; it determines the preference for immediate over future rewards. For stochastic environments and policies, the goal is to maximize the expected return for either a par- ticular initial states 0 or for a distribution of initial statess 0 ∼ P(s 0 ) J(π θ )=E π R 0 =E π ∞ X t=0 γ t r t , (A.2) PolicyGradientMethods Our goal is to maximizeJ with respect toθ . One important consideration in this context is that for most cases of interest even the exact evaluation ofJ is computationally in- tractable due to the high cost of computing the high-dimensional integral or sum required for evaluating the expectation. A number of different approaches have been developed to maximize (A.2) approximately. Here, we focus on a particular class of methods that directly approximate the gradient ofJ . In particular, using the identity∇ θ π θ =π θ ∇ θ logπ θ , which is also known as the “likelihood ratio trick”, and noting that logπ θ (τ )=logP(s 0 )+ P t=0 [logπ θ (a t |o t )+logP(s t+1 |s t ,a t )], we can express the gradient of (A.2) as follows: ∇ θ J(θ )=E π " ∞ X t=0 γ t R t ∇ θ logπ θ (a t |s t ) # . (A.3) 98 The basic intuition underlying this gradient estimate is that we consider different actions a t that the pol- icy could take in state s t ; we observe returns R t associated with these actions; and we then adjust the probability of actions according to the associated returns. Once we have an estimate of∇ θ J we can use this to update the policy parametersθ new =θ old +δ ∇ θ J . The resulting class of algorithms is named policy gradient method [137, 120, 57, 105, 106], because it com- putes a gradient of the long-horizon objective with respect to the policy for learning. Policy gradient methods are widely used to learn complex control tasks and are often regarded as the most effective rein- forcement learning techniques, especially for robotics applications [106]. They are scalable to high dimen- sional continuous state and action problems, and are guaranteed to converge to a locally optimal policy [120, 108]. However, these algorithms are rarely built on the form presented in Eq. (??). The reason is that without further modification the gradient estimate has extremely high variance and will be highly sensitive to the choice of gradient step size. In the next section we will discuss several modifications that lead to more workable solutions. Actor-Critic Methods To improve data efficiency and stability of the core policy gradient algorithm outlined above it is helpful to introduce two new functions, the state and state-action value functions usually denoted asV andQ, respectively. The state value functionV gives the discounted sum of future rewards that the agent expects to receive when following policyπ from states t , V π (s t )=E π R t s t ]=E π ∞ X t ′ =t γ t ′ − t r t s t ], (A.4) while the state-action value function Q, simply referred to as the Q-function, expresses the same quan- tity, assuming that the agent starts in state s t and selects action a t , and only thereafter chooses actions according toπ , Q π (s t ,a t )=E π R t |s t ,a t =E π ∞ X t ′ =t γ t ′ − t r t s t ,a t ]. (A.5) 99 The dependence of these functions onπ is indicated by the superscriptV π andQ π . The two functionsV π andQ π are related throughV π (s)=E π [Q π (s,a)|s]. Given A.4 and A.5 and the properties of conditional expectations, it is possible to express the gradient derived in (A.3) in terms ofV π (s t ) andQ π (s t ,a t ) (see Appendix?? for a detailed derivation) ∇ θ J(θ )=E π " ∞ X t=0 γ t Q π (s t ,a t )− V π (s t ) ∇ θ logπ θ (a t |s t ) # . (A.6) This expression differs in two important ways from the one in A.3: Firstly, rather than “scoring” an action directly by the resulting sample returnR t (received along a particular trajectory) we now directly use the expected return, i.e. averaged over all future trajectories that start ats t and with actiona t . Secondly, we further subtract the expected future return V π (s t ) that policy π would receive when starting at s t . The termV π (s t ) acts as abaseline: rather than looking at the absolute (expected) return, we only consider how much better a given action is compared to what the policy would receive on “average”. That is to say, the functionsQ π (s t ,a t ) andV π (s t ) play the respective roles of scoring actions and acting as a reference, and therefore, they are often referred to ascritic andbaseline, respectively. Due to this particular interpretation, the difference A π (s t ,a t ) = Q π (s t ,a t )− V π (s t ) is often referred to as the advantage. It is important to note thatV π (s t ) is only one of many possible choices of baseline (see Appendix ). A simple rule of thumb is that any function that does not, directly or indirectly, depend ona t is acceptable, but that some may be more appropriate than others [135, 138, 34, 142]. While A.3 and A.6 merely express the same quantity in two different ways, when used as part of a Monte Carlo estimate the modification in (A.6) becomes relevant. Given access to Q π andV π , the Monte Carlo estimates of A.6 ∇ θ J(θ )≈ 1 M M X j=1 " ∞ X t=0 γ t (Q π (s j t ,a j t )− V π (s j t ))∇ θ logπ θ (a j t |s j t ) # , (A.7) 100 Algorithm1 Environment Simulation 1: for time stept=0,1,... do 2: if t=0or episode terminatesthen 3: store time step of episode termination, 4: reset states t ∼ P(s 0 ) 5: evaluate observation: o t ∼ o(s t ) 6: endif 7: sample action from policya t ∼ π θ (a t |o t ) 8: evolve next state according to fish physics s t+1 ∼ P(s t+1 |s t ,a t ) 9: evaluate next observationo t+1 ∼ o(s t+1 ) and rewardr t ∼ r(a t ,o t+1 ) 10: if t=0or mod(t,N)̸=0then 11: append current action, observation, reward, and probability of sampling the action to assemble vectors a N× na ,o N× no ,r N× 1 , andπ θ old (a|o) N× 1 12: else 13: update agent networks according to Algorithm 2 14: endif 15: endfor is likely to have significantly lower variance than that of A.3 in ?? for the two reasons mentioned above. Namely, the sample estimate R j t (which is affected by the randomness along the trajectory τ j following s j t ) is replaced with the expectation, and the baseline is subtracted. The latter intuitively accounts for the fact that the absolute valueQ π (s t ,a t ) for a given action has little meaning; what matters is the value of an actiona t relative to other action choices possible in that same state. As a side note, we mention that in practice, the termγ t that scales the gradient in A.7) is usually dropped. Proximal Policy Optimization (PPO) Algorithms We implement the clipped advantage Proximal Policy Optimization (PPO) method proposed by [113] for our RL training. PPO maximizes a surrogate ob- jective that clips off unwanted changes when the policy deviates too much from the policy of the previous cycle to ensure faster and more robust convergence. We refer readers to the original reference cited above as well as the OpenAI’s documentation of the PPO algorithm and their baseline implementations for a thorough explanation of the theory and details behind this method. 101 Algorithm2 Updating the Agent 1: for update epoch numberκ =0,1,...K do 2: compute the truncated return using rewardsr N× 1 and assemble into vectorR N× 1 3: estimate infinite-horizon return using R N× 1 and V T = V ϕ (o T ) if bootstrapping is desired (see Eq.??) 4: using o N× no and value function V ϕ , evaluate expected returns at each time step and store into V N× 1 5: compute the advantageA=R N× 1 − V N× 1 and normalize by its mean and variance if desired 6: evaluate the probability of realizing a N× na based on o N× no for the policy π θ , and store to π θ (a|o) N× 1 7: compute the action-likelihood ratio: ϱ θ = π θ (a|o) N× 1 π θ old (a|o) N× 1 8: compute clipped surrogate loss function:L clip (θ )=mean[min[ϱ θ · A, clip(ϱ θ ,1− ϵ, 1+ϵ )· A]] 9: compute the value-function loss:L value (ϕ )=0.5· mean (R N× 1 − V N× 1 ) 2 10: compute the total loss:L(θ,ϕ )=−L clip (θ )+L value (ϕ )− α · entropy[π θ ] 11: update parameters (θ,ϕ ) to minimize the total loss using a gradient based optimizer (e.g., Adam [68]) 12: endfor 102 AppendixB Self-organizationofflow-coupledswimmersintospatialpatterns B.1 CFDmodel In our CFD simulations, a swimmer is modeled as a symmetric 2D Joukowsky airfoil [128]. The chord length of the airfoil is the characteristic lengthL, and the maximum thickness is0.12L. The airfoil under- goes pitching motion around the leading edge of the airfoil. Fluid-structure interactions are governed by the incompressible Navier-Stokes equations, ∇· u=0, ∂u ∂t +u·∇u=−∇ p+ 1 Re ∆ u, (B.1) where u(x,t) snf p(x,t) are the velocity and pressure field, respectively. These fields are solved using immersed boundary method (IBM) that handles the two way coupled fluid structure interaction [104, 35, 37, 11, 36, 91]. The immersed boundary formulation involves an Eulerian descriptions of the flow field and a La- grangian description of the immersed airfoils. The boundary condition is mapped to a body force exerted on the fluid. The Lagrangian and Eulerian variables are correlated by the Dirac delta function, which is smoothed when performing discretization. Here, we used the implementation developed by the group of Professor Boyce Griffith, IBAMR [48], which has long been used to solve problems such as blood flow 103 80L 20L L L d(t) 10L θ 1 (t) θ 2 (t) A B L.E. s L.E. M Figure B.1: CFD simulation setup for a pair of swimmers. A. shows the computational domain and schematics of the schooling airfoils. B. shows the hydrodynamic forces acting on the airfoil. It is composed of pressure force p(s) and wall stress force τ (s), in whichs is the curvilinear coordinate along the body starting from leading edge of the airfoil and goes clockwise. After integration, we can getFx in swimming direction,Fy in lateral direction, andM is the moment acting at leading edge. in heart [104, 73], water entry/exit problems [12], fish’s swimming [45, 129, 134], insect’s flight [131, 132], flexible propulsors [44, 130, 43], self propulsion of pitching/heaving airfoil [43, 139], and fish school- ing [139, 82]. This implementation is based on an adaptive mesh, which enables us to accurately simulate self propulsion cases and reach steady state in a large computational domain with a reasonable computa- tional cost. The setup of the simulation is given as in Figure B.1. The computational domain is chosen to be a rectangular 80L× 20L. We used periodic boundary conditions on the boundary of the computational domain and no-slip boundary condition on the surface of airfoils. The initial location of the leader is12L away from the right boundary, and the follower is behind it. The initial distance between the two swimmer d(t=0) ranges from1.5L to4L to make the follower falls into different equilibrium in the wake of leader. The coarsest Eulerian mesh is a uniform600× 125 Cartesian grid. There are2 layers of refinement mesh, and the refinement ratio for each layer is 4. Both the computational region close to the fish and inside the wake are refined. The simulation timestep is adaptive, with the maximum value ∆ t max =2.5× 10 − 3 . 104 B.2 Time-delayparticlemodel We study the pairwise interaction of flow-coupled oscillators in the context of the minimal particle model used in [9, 94]. This particle model was designed for inline swimmers. Here, we modify it slightly to account for lateral offset ℓ between the swimmers. In a nutshell, the model assumes that the leader leaves behind a vertical wake speed that is equal to the leader’s flapping speed at the tail. The speed of the ‘wake’ that is left behind decays exponentially in time, as an approximation for viscous dissipation. Details of the model can be found in the Supplementary Information of [94]. Considering two swimmers flapping vertically according to y 1 = 1 2 A 1 sin(2πf 1 t), y 2 = 1 2 A 2 sin(2πf 2 t− ϕ ), (B.2) Each particle is assumed to experience a thrust force F j that is proportional to the square of its vertical velocity relative to the surrounding fluid, and a drag force D j relative to its relative horizontal speed. Namely, m¨x j =− F j +D j , =ρL − C T (˙ y j − v(x j ,y j )) 2 +C D (˙ x j − u(x j ,y j )) 2 , (B.3) wherem is the mass per unit depth of the the swimmer,C T ,C D are the constant thrust and drag coeffi- cients,u,v are the horizontal and vertical components of the ambient fluid velocity, ρ is the fluid density, L the chord length. Since the leader swims into quiescent fluid, we assume that v(x 1 ,y 1 )=0. The follower swims into the wake of the leader, which we assume to have zero horizontal velocity (u=0) and vertical velocity that decays exponentially both in time and in the lateral direction, v(t,∆ t)= ˙ y 1 (t− ∆ t)e − ∆ t/τ e − (ℓ/h) p . (B.4) 105 where∆ t is the delay time between the leader and follower, that is, the time past since the leader passed the follower’s current location;t− ∆ t is the time in the past when the leader generated the wake at position x 1 (t− ∆ t). We added the last term to consider decay in the lateral directionℓ; to estimate the parameters p andh, we used a best curve fit to the data of period-average velocity magnitude versus lateral distance in the wake of a single swimmer in the vortex sheet model; see Fig. B.2A. The wake speed at the position of the follower is thus given by v(x 2 (t,∆ t))=πA 1 f 1 e − ∆ t/τ e − (l/h) p cos(2πf 1 (t− ∆ t)). (B.5) Substituting this wake speed back into Eq. (B.3), we arrive at an expression for the thrust force on the follower. We average this thrust force over a period, assumingf 1 =f 2 and∆ t is constant, ⟨F 2 ⟩=ρLC T π 2 (A 2 f 2 ) 2 +(A 1 f 1 e − ∆ t/τ e − (l/h) p ) 2 − A 2 f 2 A 1 f 1 e − (l/h) p e − ∆ t/τ cos 2πf 2 t− ϕ − 2πf 1 (t− ∆ t) (B.6) The drag experienced by the follower is proportional to its horizontal swimming speed, because the fol- lower is assumed to swim into a wake with no horizontal velocity, ⟨D 2 ⟩=ρLC D ⟨˙ x 2 ⟩ 2 (B.7) Taken together, we write the period-average equation of motion for the follower as, m⟨¨x 2 ⟩=−⟨ F 2 ⟩+⟨D 2 ⟩ (B.8) We numerically integrated Eq. (B.8) and solved for the motion of the follower. Considering a leader moving at a constant period-average swimming speed⟨˙ x 1 ⟩, we computed the separation distanced=⟨x 2 ⟩−⟨ x 1 ⟩ 106 2 1.6 1.2 0.8 0.4 0 Period-averaged flow velocity 1 0 −1 −0.5 0.5 C fitted exponential curve vortex sheet model Lateral distance , 50 40 30 20 10 0 Time, t/T 1.6 1.2 0.8 1 1.4 A Separation distance, d /UT increasing φ 1.2 1.1 1 1.05 1.15 B Separation distance, d /UT increasing 50 40 30 20 10 0 Time, t/T 1.12 1.1 1.08 1.06 1.04 1.02 Separation distance, d/UT 0.2 0 −0.2 −0.1 0.1 D Fluid force on the follower increasing Figure B.2: A. Lateral decay of flow speed in the wake of a solitary swimmer in the vortex sheet model (blue line) and the fitted exponential curve (red line). The exponential decay has has the form exp − (ℓ/1.6) 2.73 . B. Separation distance as a function of time based on the time-delay particle model for an inline pair and increasing phase lagϕ/ 2π = 0,0.125,0.25,0.5. C. Separation distance as a function of time for an inphase pair at increasing lateral distanceℓ/L=0,0.25,0.5,0.75,1. D. Fluid force experienced by the follower versus separation distance for the cases shown inC. around equilibria. between the pair acting on the follower for a range of ϕ ∈ [0,2π ] and l ∈ [0,2L] (Fig. B.2B,C). The parameter values are chosen to be consistent with the experiments of Newboltetal. (provided in Table S1 of SI in [94]). Specifically, ρ =1 g/cm 3 ,L=4 cm,m=5.3 g/cm,C D =0.25,C T =0.96,τ =0.5s. 107 B.3 Hydrodynamicpower Pitching motions are produced by an active moment M p imposed by the swimmer on the surrounding fluid about the leading edge. The value of M p is obtained from the balance of angular momentum about the swimmer’s leading edge (l.e.), I ¨θ − [m(˙ x+i˙ y)w l.e. ]=M +M p , (B.9) Here, I = m(2l) 2 /3 is the swimmer’s moment of inertia about the leading edge, w l.e. is the swimmer’s velocity at the leading edge (in complex form), and M is the hydrodynamic moment about the leading edge given in (6.11). The hydrodynamic power exerted by the swimmer onto the fluid due to its pitching motion is given by P p =M p ˙ θ. (B.10) Note the skin drag forces do not contribute to the input power. B.4 Stabilityanalysis Consider a pair of freely interacting swimmers in steady-state formation, where both swimmers have reached the same terminal speed, at a streamwise separation distanced and lateral spacingℓ. We propose to study the stability of these equilibria in the context of three models. By convention,x andd increase to the right (positivex-axis) while the swimmers move to the left. In this convention, drag forces are positive and thrust forces are negative. Linear stability analysis of pairwise formation 108 Figure B.3: Swimming power is in pairwise formation. A. Swimming speed andC. input power of a solitary freely swimming oscillator and B. swimming speed and D. input power of a pair of inline and inphase oscillators. The blue and red lines denote the leader and follower, respectively. The dashed lines show the period-average input power for each case. The results are shown during 5 periods in the steady state. The pair swimming speed is1.7% slower than the solitary swimmer. The leader uses0.5% more power and the follower saves65.8% relative to a solitary swimmer. We introduce a perturbation to the formation; see Fig. B.4. Namely, we perturb the position of the follower about the equilibrium in the x-direction with a perturbation of size δd/L = δx/L = 0.5 and we calculate the corresponding change in distanceδd and change in total hydrodynamic forceδF x acting on the follower in the x-direction. These quantities are shown over time in Fig. B.4A,B. Blue and black lines indicate instantaneous and period-average values, respectively. We incrementally sample the period- averageδd andδF x over a window of time after the perturbation, plot the sampled forceδF x as a function of perturbation δd , and fit a straight line through the data (Fig. B.4C). The hydrodynamic force acts as a restoring force δF x = − Kδd , bringing the follower back to its equilibrium distance from the leader. PositiveK implies that the formation is stable. Furthermore, the magnitude ofK provides a measure of how stable and cohesive the formation is, which we show in Fig. 3C of the main text for all formations. Stability analysis based on virtual particle in the wake of solitary swimmer We consider a freely swimming follower placed at a location(x o ,y o ) in the wake of a flapping swimmer. At steady state, the pressure forceF x (x o ,y o ,t) and skin dragD(x o ,y o ,t) balance each other on average, 109 −0.5 −0.25 0 0.25 0.5 50 45 40 35 30 55 60 −12 −8 0 4 12 −1 0 1 2 8 −4 hydrodynamic force Hydrodynamic force Perturbation pairwise cohesion A C Time, t/T B 50 45 40 35 30 55 60 −0.25 0.25 0 Time, t/T Perturbation Figure B.4: Emergent pairwise formations are stable to perturbations in separation distance. A. Separation distance between swimmers at steady state is perturbed att = 40T . B. the hydrodynamic force on the follower as a function of time (blue and black show instantaneous and period-average values, respectively) as the follower position is perturbed in thex direction at time t = 40T . The hydrodynamic force on the follower acts as a restorative force that decays the perturbation back to an average value of zero. C. Hydrodynamic force and distance perturbations sampled formA. andB. at increments of 1 per period, over 20 periods after the perturbation. We plot the line with the best fit to the data points whose slope is a measure of the cohesiveness of the formation. and the follower experiences zero net acceleration. Namely,−⟨ F x (x o ,y o )⟩+⟨D(x o ,y o )⟩ = 0, where the time average notation⟨(·)⟩=(1/T) R t+T t (·)dt ′ is introduced for brevity. If we perturb the horizontal position of the follower byδx , since skin drag depends only on the relative fluid’s velocity tangential to the plate, it is reasonable to assume that its change due to in-line positional perturbations is negligible⟨D(x o +δx,y o )⟩≈⟨ D(x o ,y o )⟩. We thus arrive at the period-average equation −⟨ F x (x o +δx,y o )⟩+⟨F(x o ,y o )⟩=m⟨δ ¨x⟩. This equation provides a condition for the linear stability of the pairwise formation: if the slope of the period-average force relative to the horizontal positionδx is neg- ative, the system is linearly stable to perturbations in the horizontal position. Otherwise, the perturbation grows and the pair leaves their relative spacing at steady state. We use this understanding to propose a criterion for checking the linear stability of a virtual particle follower located at (x o ,y o ) in the flow field u created by a solitary swimmer. The flow field u of the solitary swimmer is obtained here from CFD or VS models, but the approach can be readily applied to flows obtained from experimental Particle Image Velocimetry (PIV) measurements. Considering the virtual follower is undergoing vertical oscillationsy(t)=y o +Asin(2πt − ϕ ), whereA is the oscillation amplitude 110 andϕ is the phase lag of the follower, with associated velocity vectorv, we define a measure of the period- average thrustT acting on the virtual follower T(x o ,y o ,ϕ )=− 1 T R t+T t ((v− u).e y ) 2 dt ′ 1 T R t+T t (2v.e y ) 2 dt ′ . (B.11) It thus suffices to obtain the slope of T with respect toδx to gauge the stability of the formation: negative slope∂T/∂x predicts stable formation. We refer to the slope of∂T/∂x as the cohesion parameter. In Fig. 4D of the main text we show thrust parameter as a function of the separation distance in the wake of a solitary swimmer in the CFD and VS model, for three different lateral distances ℓ. The dashed lines around d/UT =1,2 mark the positions where the follower stabilizes at in the pairwise interaction. The slopes at these positions are negative, implying the stability of the formations. We plot the magnitude of the slope of the thrust parameter at these locations as a function of lateral distance in Fig. 4F. The slope decay as a function ofℓ, implying that the cohesion of the formations decay with increasing lateral distance. Stability analysis based on time-delay particle model We seek a relationship between pairwise cohesion and lateral distanceℓ in the context of the time-delay particle model. For an inphase pair, starting at initial distance d/UT = 1.15, we incrementally increase the lateral offset from ℓ = 0 to ℓ = L. The separation distance is shown as a function of time in Fig. B.2C. Due to the decay of the leader’s wake in the lateral direction (see Eq. B.4), the hydrodynamic force experienced by the follower decreases at increasing lateral offset ℓ. We plot the hydrodynamic force⟨F 2 ⟩ acting on the follower given in (B.6) and the corresponding distance around the equilibrium in Fig. B.2D. Here, we define our measure of stability similar to our stability analysis in the pairwise formation and the in the wake of a solitary swimmer model. Namely, we compute the slope of force versus distanceδF/δd (shown by orange lines in Fig. B.2D) as a measure of linear stability. The slopes are negative for all the lateral 111 distances, implying that the formations are stable, however the magnitudes of slopes decay as we increase ℓ. 112
Abstract (if available)
Abstract
Collective motion is widely observed in the animal kingdom from schooling fish to cooperative transport in ants. The underlying mechanism that give rise to collective motion are not always clear. Growing evidence suggest that the interaction of individual group members with the physical environment plays a key role in the emergence of collective behavior.
In this thesis, we explore this hypothesis in the context of two distinct model systems: (i) cooperative locomotion in tube feet inspired actuators and (ii) self-organization of flow-coupled swimmers.
In the first part of this thesis, we propose a reduced-order model of the sea star body and tube feet biomechanics. In this model, there is no communication of sensory information between individual tube feet, but they are coupled mechanically through their connection to a rigid body. We find that these minimally coupled tube feet coordinate to generate robust locomotion on various terrain.
In the second part, we study the emergent dynamics of flow-coupled swimmers in the context of the vortex sheet model. We demonstrate that flow interactions lead to the emergence of versatile formations of various spatial patterns, ranging from cooperative patterns that favor fair distribution of energy savings among group members to greedy patterns where few members get maximal benefit, leaving trailing swimmers with diminished opportunities for gaining energetic benefits.
Our findings emphasize the role of the physical environment in the emergence of cooperative locomotion in biology. They also offer a new paradigm for locomotion using minimally-coupled actuators with potential applications to autonomous robotic systems.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Evaluating sensing and control in underwater animal behaviors
PDF
Synchronization in carpets of model cilia: numerical simulations and continuum theory
PDF
Traveling sea stars: hydrodynamic interactions and radially-symmetric motion strategies for biomimetic robot design
PDF
Transitions in the collective behavior of microswimmers confined to microfluidic chips
PDF
Passive flight in density-stratified fluid environments
PDF
Physics-based and data-driven models for bio-inspired flow sensing and motion planning
PDF
An experimental study of the elastic theory of granular flows
PDF
Supersonic flow study of a capsule geometry using large-eddy simulation
PDF
Dynamic modeling and simulation of flapping-wing micro air vehicles
PDF
Flow and thermal transport at porous interfaces
PDF
Multiscale modeling of cilia mechanics and function
PDF
Passive rolling and flapping dynamics of a heaving Λ flyer
PDF
Reconstruction and estimation of shear flows using physics-based and data-driven models
PDF
Pattern generation in stratified wakes
PDF
Hydrodynamic stability of two fluid columns of different densities and viscosities subject to gravity
PDF
Wing selection and design for flapping flight at small scale
PDF
Numerical study of shock-wave/turbulent boundary layer interactions on flexible and rigid panels with wall-modeled large-eddy simulations
PDF
Synergistic coupling between geomechanics, flow, and transport in fractured porous media: applications in hydraulic fracturing and fluid mixing
PDF
Numerical study of flow characteristics of controlled vortex induced vibrations in cylinders
PDF
Development and applications of a body-force propulsor model for high-fidelity CFD
Asset Metadata
Creator
Heydari, Sina
(author)
Core Title
Mechanical and flow interactions facilitate cooperative transport and collective locomotion in animal groups
School
Viterbi School of Engineering
Degree
Doctor of Philosophy
Degree Program
Mechanical Engineering
Degree Conferral Date
2023-05
Publication Date
11/02/2023
Defense Date
03/20/2023
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
collective behavior,collective motion,fish schooling,fluid-structure interaction,OAI-PMH Harvest,reinforcement learning,sea star locomotion,tube feet,vortex sheet model
Format
theses
(aat)
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Kanso, Eva (
committee chair
), Luhar, Mitul (
committee member
), McHenry, Matthew (
committee member
), Nakano, Aiichiro (
committee member
)
Creator Email
sina.heydari1372@gmail.com,sinaheyd@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-oUC113096944
Unique identifier
UC113096944
Identifier
etd-HeydariSin-11755.pdf (filename)
Legacy Identifier
etd-HeydariSin-11755
Document Type
Dissertation
Format
theses (aat)
Rights
Heydari, Sina
Internet Media Type
application/pdf
Type
texts
Source
20230503-usctheses-batch-1035
(batch),
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Repository Email
cisadmin@lib.usc.edu
Tags
collective behavior
collective motion
fish schooling
fluid-structure interaction
reinforcement learning
sea star locomotion
tube feet
vortex sheet model