Page 236 |
Save page Remove page | Previous | 236 of 289 | Next |
|
small (250x250 max)
medium (500x500 max)
Large (1000x1000 max)
Extra Large
large ( > 500x500)
Full Resolution
All (PDF)
|
This page
All
|
where the new noise vector ˜ !t has one additional cofficient. The second equation treats
ft as another basis function whose parameter is constant and is thus simply not updated.
Thus, we added ft to the command cost instead of treating it as a state cost.
We also numerically experimented with violations of the clean distinction between
state and command cost. Equation (6.23) could be replaced by a cost term, which is
an arbitrary function of state and command. In the end, this cost term is just used to
differentiate the different roll-outs in a reward weighted average, similarly as in (Peters &
Schaal 2008a, Kober & Peters 2009). We noticed in several instances that PI2 continued
to work just fine with this improper cost formulation.
Again, it appears that the path integral formalism and the PI2 algorithm allow the
user to exploit creativity in designing cost functions, without absolute need to adhere
perfectly to the theoretical framework.
6.8.5 Dealing with hidden state
Finally, it is interesting to consider in how far PI2 would be affected by hidden state.
Hidden state can either be of stochastic or deterministic nature, and we consider hidden
state as adding additional equations to the system dynamics (4.2).
Section 4.2 already derived that deterministic hidden states drop out of the PI2
update equations – these components of the system dynamics were termed “uncontrolled”
equations.
More interesting are hidden state variables that have stochastic differential equations,
i.e., these equations are uncontrolled but do have a noise term and a non-zero corre-sponding
coefficient in Gt in equation (4.2), and these equations are coupled to the other
222
Object Description
| Title | Iterative path integral stochastic optimal control: theory and applications to motor control |
| Author | Theodorou, Evangelos A. |
| Author email | etheodor@usc.edu; theo0027@umn.edu |
| Degree | Doctor of Philosophy |
| Document type | Dissertation |
| Degree program | Computer Science |
| School | Viterbi School of Engineering |
| Date defended/completed | 2011-01-11 |
| Date submitted | 2011 |
| Restricted until | Unrestricted |
| Date published | 2011-04-29 |
| Advisor (committee chair) | Schaal, Stefan |
| Advisor (committee member) |
Valero-Cuevas, Francisco Sukhatme, Gaurav S. Todorov, Emo Schweighofer, Nicolas |
| Abstract | Motivated by the limitations of current optimal control and reinforcement learning methods in terms of their efficiency and scalability, this thesis proposes an iterative stochastic optimal control approach based on the generalized path integral formalism. More precisely, we suggest the use of the framework of stochastic optimal control with path integrals to derive a novel approach to RL with parameterized policies. While solidly grounded in value function estimation and optimal control based on the stochastic Hamilton Jacobi Bellman (HJB) equation, policy improvements can be transformed into an approximation problem of a path integral which has no open algorithmic parameters other than the exploration noise. The resulting algorithm can be conceived of as model-based, semi-model-based, or even model free, depending on how the learning problem is structured. The new algorithm, Policy Improvement with Path Integrals (PI2), demonstrates interesting similarities with previous RL research in the framework of probability matching and provides intuition why the slightly heuristically motivated probability matching approach can actually perform well. Applications to high dimensional robotic systems are presented for a variety of tasks that require optimal planning and gain scheduling.; In addition to the work on generalized path integral stochastic optimal control, in this thesis we extend model based iterative optimal control algorithms to the stochastic setting. More precisely we derive the Differential Dynamic Programming algorithm for stochastic systems with state and control multiplicative noise. Finally, in the last part of this thesis, model based iterative optimal control methods are applied to bio-mechanical models of the index finger with the goal to find the underlying tendon forces applied for the movements of, tapping and flexing. |
| Keyword | stochastic optimal control; reinforcement learning,; robotics |
| Language | English |
| Part of collection | University of Southern California dissertations and theses |
| Publisher (of the original version) | University of Southern California |
| Place of publication (of the original version) | Los Angeles, California |
| Publisher (of the digital version) | University of Southern California. Libraries |
| Provenance | Electronically uploaded by the author |
| Type | texts |
| Legacy record ID | usctheses-m3804 |
| Contributing entity | University of Southern California |
| Rights | Theodorou, Evangelos A. |
| Repository name | Libraries, University of Southern California |
| Repository address | Los Angeles, California |
| Repository email | cisadmin@lib.usc.edu |
| Filename | etd-Theodorou-4581 |
| Archival file | uscthesesreloadpub_Volume14/etd-Theodorou-4581.pdf |
Description
| Title | Page 236 |
| Contributing entity | University of Southern California |
| Repository email | cisadmin@lib.usc.edu |
| Full text | where the new noise vector ˜ !t has one additional cofficient. The second equation treats ft as another basis function whose parameter is constant and is thus simply not updated. Thus, we added ft to the command cost instead of treating it as a state cost. We also numerically experimented with violations of the clean distinction between state and command cost. Equation (6.23) could be replaced by a cost term, which is an arbitrary function of state and command. In the end, this cost term is just used to differentiate the different roll-outs in a reward weighted average, similarly as in (Peters & Schaal 2008a, Kober & Peters 2009). We noticed in several instances that PI2 continued to work just fine with this improper cost formulation. Again, it appears that the path integral formalism and the PI2 algorithm allow the user to exploit creativity in designing cost functions, without absolute need to adhere perfectly to the theoretical framework. 6.8.5 Dealing with hidden state Finally, it is interesting to consider in how far PI2 would be affected by hidden state. Hidden state can either be of stochastic or deterministic nature, and we consider hidden state as adding additional equations to the system dynamics (4.2). Section 4.2 already derived that deterministic hidden states drop out of the PI2 update equations – these components of the system dynamics were termed “uncontrolled” equations. More interesting are hidden state variables that have stochastic differential equations, i.e., these equations are uncontrolled but do have a noise term and a non-zero corre-sponding coefficient in Gt in equation (4.2), and these equations are coupled to the other 222 |
