Page 215 |
Save page Remove page | Previous | 215 of 289 | Next |
|
small (250x250 max)
medium (500x500 max)
Large (1000x1000 max)
Extra Large
large ( > 500x500)
Full Resolution
All (PDF)
|
This page
All
|
should be emphasized that there were absolutely no parameter tuning needed to achieve
the PI2 results, while all gradient algorithms required readjusting of learning rates for
every example to achieve best performance.
6.4.4 Application to robot learning
Figure 6.4 illustrates our application to a robot learning problem. The robot dog is to
jump across as gap. The jump should make forward progress as much as possible, as it is
a maneuver in a legged locomotion competition which scores the speed of the robot – note
that we only used a physical simulator of the robot for this experiment, as the actual robot
was not available. The robot has three DOFs per leg, and thus a total of d = 12 DOFs.
Each DOF was represented as a DMP with 50 basis functions. An initial seed behavior
(Figure 6.5-top) was taught by learning from demonstration, which allowed the robot
barely to reach the other side of the gap without falling into the gap – the demonstration
was generated from a manual adjustment of spline nodes in a spline-based trajectory plan
for each leg.
PI2 learning used primarily the forward progress as a reward, and slightly penalized
the squared acceleration of each DOF, and the length of the parameter vector. Addi-tionally,
a penalty was incurred if the yaw or the roll exceeded a threshold value – these
penalties encouraged the robot to jump straight forward and not to the side, and not to
fall over. The exact cost function is:
rt = rroll + ryaw +
>d
i=1
A
a1f2
i,t + 0.5a2 )Ti
)
B
(a1 = 1.e − 6, a2 = 1.e − 8) (6.34)
201
Object Description
| Title | Iterative path integral stochastic optimal control: theory and applications to motor control |
| Author | Theodorou, Evangelos A. |
| Author email | etheodor@usc.edu; theo0027@umn.edu |
| Degree | Doctor of Philosophy |
| Document type | Dissertation |
| Degree program | Computer Science |
| School | Viterbi School of Engineering |
| Date defended/completed | 2011-01-11 |
| Date submitted | 2011 |
| Restricted until | Unrestricted |
| Date published | 2011-04-29 |
| Advisor (committee chair) | Schaal, Stefan |
| Advisor (committee member) |
Valero-Cuevas, Francisco Sukhatme, Gaurav S. Todorov, Emo Schweighofer, Nicolas |
| Abstract | Motivated by the limitations of current optimal control and reinforcement learning methods in terms of their efficiency and scalability, this thesis proposes an iterative stochastic optimal control approach based on the generalized path integral formalism. More precisely, we suggest the use of the framework of stochastic optimal control with path integrals to derive a novel approach to RL with parameterized policies. While solidly grounded in value function estimation and optimal control based on the stochastic Hamilton Jacobi Bellman (HJB) equation, policy improvements can be transformed into an approximation problem of a path integral which has no open algorithmic parameters other than the exploration noise. The resulting algorithm can be conceived of as model-based, semi-model-based, or even model free, depending on how the learning problem is structured. The new algorithm, Policy Improvement with Path Integrals (PI2), demonstrates interesting similarities with previous RL research in the framework of probability matching and provides intuition why the slightly heuristically motivated probability matching approach can actually perform well. Applications to high dimensional robotic systems are presented for a variety of tasks that require optimal planning and gain scheduling.; In addition to the work on generalized path integral stochastic optimal control, in this thesis we extend model based iterative optimal control algorithms to the stochastic setting. More precisely we derive the Differential Dynamic Programming algorithm for stochastic systems with state and control multiplicative noise. Finally, in the last part of this thesis, model based iterative optimal control methods are applied to bio-mechanical models of the index finger with the goal to find the underlying tendon forces applied for the movements of, tapping and flexing. |
| Keyword | stochastic optimal control; reinforcement learning,; robotics |
| Language | English |
| Part of collection | University of Southern California dissertations and theses |
| Publisher (of the original version) | University of Southern California |
| Place of publication (of the original version) | Los Angeles, California |
| Publisher (of the digital version) | University of Southern California. Libraries |
| Provenance | Electronically uploaded by the author |
| Type | texts |
| Legacy record ID | usctheses-m3804 |
| Contributing entity | University of Southern California |
| Rights | Theodorou, Evangelos A. |
| Repository name | Libraries, University of Southern California |
| Repository address | Los Angeles, California |
| Repository email | cisadmin@lib.usc.edu |
| Filename | etd-Theodorou-4581 |
| Archival file | uscthesesreloadpub_Volume14/etd-Theodorou-4581.pdf |
Description
| Title | Page 215 |
| Contributing entity | University of Southern California |
| Repository email | cisadmin@lib.usc.edu |
| Full text | should be emphasized that there were absolutely no parameter tuning needed to achieve the PI2 results, while all gradient algorithms required readjusting of learning rates for every example to achieve best performance. 6.4.4 Application to robot learning Figure 6.4 illustrates our application to a robot learning problem. The robot dog is to jump across as gap. The jump should make forward progress as much as possible, as it is a maneuver in a legged locomotion competition which scores the speed of the robot – note that we only used a physical simulator of the robot for this experiment, as the actual robot was not available. The robot has three DOFs per leg, and thus a total of d = 12 DOFs. Each DOF was represented as a DMP with 50 basis functions. An initial seed behavior (Figure 6.5-top) was taught by learning from demonstration, which allowed the robot barely to reach the other side of the gap without falling into the gap – the demonstration was generated from a manual adjustment of spline nodes in a spline-based trajectory plan for each leg. PI2 learning used primarily the forward progress as a reward, and slightly penalized the squared acceleration of each DOF, and the length of the parameter vector. Addi-tionally, a penalty was incurred if the yaw or the roll exceeded a threshold value – these penalties encouraged the robot to jump straight forward and not to the side, and not to fall over. The exact cost function is: rt = rroll + ryaw + >d i=1 A a1f2 i,t + 0.5a2 )Ti ) B (a1 = 1.e − 6, a2 = 1.e − 8) (6.34) 201 |
