Page 140 |
Save page Remove page | Previous | 140 of 289 | Next |
|
small (250x250 max)
medium (500x500 max)
Large (1000x1000 max)
Extra Large
large ( > 500x500)
Full Resolution
All (PDF)
|
This page
All
|
4.2 Generalized path integral formalism
To develop our algorithms, we will need to consider a more general development of the
path integral approach to stochastic optimal control than presented in (Kappen 2007) and
(Broek et al. 2008). In particular, we have to address that in many stochastic dynamical
systems, the control transition matrix Gt is state dependent and its structure depends on
the partition of the state in directly and non-directly actuated parts. Since only some of
the states are directly controlled, the state vector is partitioned into x = [x(m)T
x(c)T
]T
with x(m) ! "k×1 the non-directly actuated part and x(c) ! "l×1the directly actuated
part. Subsequently, the passive dynamics term and the control transition matrix can be
partitioned as ft = [f (m)
t
T
f (c)
t
T
]T with fm ! "k×1, fc ! "l×1 and Gt = [0k×p G(c)
t
T
]T
with G(c)
t ! "l×p. The discretized state space representation of such systems is given as:
xti+1 = xti + ftidt +Gtiutidt + Btidwti ,
or, in partitioned vector form:
8
99:
x(m)
ti+1
x(c)
ti+1
;
<<=
=
8
99:
x(m)
ti
x(c)
ti
;
<<=
+
8
99:
f (m)
ti
f (c)
ti
;
<<=
dt +
8
99:
0k×p
G(c)
ti
;
<<=
utidt +
8
99:
0k×p
B(c)
ti
;
<<=
dwti . (4.18)
Essentially the stochastic dynamics are partitioned into controlled equations in which
the state x(c)
ti+1 is directly actuated and the uncontrolled equations in which the state x(m)
ti+1
126
Object Description
| Title | Iterative path integral stochastic optimal control: theory and applications to motor control |
| Author | Theodorou, Evangelos A. |
| Author email | etheodor@usc.edu; theo0027@umn.edu |
| Degree | Doctor of Philosophy |
| Document type | Dissertation |
| Degree program | Computer Science |
| School | Viterbi School of Engineering |
| Date defended/completed | 2011-01-11 |
| Date submitted | 2011 |
| Restricted until | Unrestricted |
| Date published | 2011-04-29 |
| Advisor (committee chair) | Schaal, Stefan |
| Advisor (committee member) |
Valero-Cuevas, Francisco Sukhatme, Gaurav S. Todorov, Emo Schweighofer, Nicolas |
| Abstract | Motivated by the limitations of current optimal control and reinforcement learning methods in terms of their efficiency and scalability, this thesis proposes an iterative stochastic optimal control approach based on the generalized path integral formalism. More precisely, we suggest the use of the framework of stochastic optimal control with path integrals to derive a novel approach to RL with parameterized policies. While solidly grounded in value function estimation and optimal control based on the stochastic Hamilton Jacobi Bellman (HJB) equation, policy improvements can be transformed into an approximation problem of a path integral which has no open algorithmic parameters other than the exploration noise. The resulting algorithm can be conceived of as model-based, semi-model-based, or even model free, depending on how the learning problem is structured. The new algorithm, Policy Improvement with Path Integrals (PI2), demonstrates interesting similarities with previous RL research in the framework of probability matching and provides intuition why the slightly heuristically motivated probability matching approach can actually perform well. Applications to high dimensional robotic systems are presented for a variety of tasks that require optimal planning and gain scheduling.; In addition to the work on generalized path integral stochastic optimal control, in this thesis we extend model based iterative optimal control algorithms to the stochastic setting. More precisely we derive the Differential Dynamic Programming algorithm for stochastic systems with state and control multiplicative noise. Finally, in the last part of this thesis, model based iterative optimal control methods are applied to bio-mechanical models of the index finger with the goal to find the underlying tendon forces applied for the movements of, tapping and flexing. |
| Keyword | stochastic optimal control; reinforcement learning,; robotics |
| Language | English |
| Part of collection | University of Southern California dissertations and theses |
| Publisher (of the original version) | University of Southern California |
| Place of publication (of the original version) | Los Angeles, California |
| Publisher (of the digital version) | University of Southern California. Libraries |
| Provenance | Electronically uploaded by the author |
| Type | texts |
| Legacy record ID | usctheses-m3804 |
| Contributing entity | University of Southern California |
| Rights | Theodorou, Evangelos A. |
| Repository name | Libraries, University of Southern California |
| Repository address | Los Angeles, California |
| Repository email | cisadmin@lib.usc.edu |
| Filename | etd-Theodorou-4581 |
| Archival file | uscthesesreloadpub_Volume14/etd-Theodorou-4581.pdf |
Description
| Title | Page 140 |
| Contributing entity | University of Southern California |
| Repository email | cisadmin@lib.usc.edu |
| Full text | 4.2 Generalized path integral formalism To develop our algorithms, we will need to consider a more general development of the path integral approach to stochastic optimal control than presented in (Kappen 2007) and (Broek et al. 2008). In particular, we have to address that in many stochastic dynamical systems, the control transition matrix Gt is state dependent and its structure depends on the partition of the state in directly and non-directly actuated parts. Since only some of the states are directly controlled, the state vector is partitioned into x = [x(m)T x(c)T ]T with x(m) ! "k×1 the non-directly actuated part and x(c) ! "l×1the directly actuated part. Subsequently, the passive dynamics term and the control transition matrix can be partitioned as ft = [f (m) t T f (c) t T ]T with fm ! "k×1, fc ! "l×1 and Gt = [0k×p G(c) t T ]T with G(c) t ! "l×p. The discretized state space representation of such systems is given as: xti+1 = xti + ftidt +Gtiutidt + Btidwti , or, in partitioned vector form: 8 99: x(m) ti+1 x(c) ti+1 ; <<= = 8 99: x(m) ti x(c) ti ; <<= + 8 99: f (m) ti f (c) ti ; <<= dt + 8 99: 0k×p G(c) ti ; <<= utidt + 8 99: 0k×p B(c) ti ; <<= dwti . (4.18) Essentially the stochastic dynamics are partitioned into controlled equations in which the state x(c) ti+1 is directly actuated and the uncontrolled equations in which the state x(m) ti+1 126 |
