Page 153 |
Save page Remove page | Previous | 153 of 289 | Next |
|
small (250x250 max)
medium (500x500 max)
Large (1000x1000 max)
Extra Large
large ( > 500x500)
Full Resolution
All (PDF)
|
This page
All
|
However, we will leave the discussion for PI2 for the next chapter and in this section we
present the general version of iterative path integral control.
In particular, we start by looking into the expectation (4.15) in the Feynman Kac
Lemma that is evaluated over the trajectories " i =
A
xti , xti+1, ....., xtN
B
sampled with
the forward propagation of uncontrolled diffusion dx = f (x, t)dt + B(x, t)Ldw. This
sampling approach is inefficient since it is very likely that parts of the state space relevant
to the optimal control task may not be reached by the sampled trajectories at once. In
addition, it has poor scalability properties when applied to high dimensional robotic
optimal control problems. Besides the reason of poor sampling, it is very common in
robotics applications to have an initial controller-policy which is manually tuned and
found based on experience. In such cases, the goal is to improve this initial policy by
performing an iterative process. At every iteration (i) the policy *u(i−1) is applied to
the dynamical system to generate state space trajectories which are going to be used for
improving the current policy. The policy improvement results from the evaluation of the
expectation (4.16) of the Feynman - Kac Lemma on the sampled trajectories and the use
of the path integral control formalism to find *u(i). The old policy *u(i−1) is updated
according to *u(i−1) + *u(i) and the process repeats again with the generation of the
new state space trajectories according to the updated policy. In mathematical terms the
iterative version of Path Integral Control is expressed as follows:
V (i)(x) = min
$u(i)
J(x, u) = min
$u(i)
!" tN
to
?
q(x, t) + *u(i)T R *u(i)
@
dt
#
(4.36)
139
Object Description
| Title | Iterative path integral stochastic optimal control: theory and applications to motor control |
| Author | Theodorou, Evangelos A. |
| Author email | etheodor@usc.edu; theo0027@umn.edu |
| Degree | Doctor of Philosophy |
| Document type | Dissertation |
| Degree program | Computer Science |
| School | Viterbi School of Engineering |
| Date defended/completed | 2011-01-11 |
| Date submitted | 2011 |
| Restricted until | Unrestricted |
| Date published | 2011-04-29 |
| Advisor (committee chair) | Schaal, Stefan |
| Advisor (committee member) |
Valero-Cuevas, Francisco Sukhatme, Gaurav S. Todorov, Emo Schweighofer, Nicolas |
| Abstract | Motivated by the limitations of current optimal control and reinforcement learning methods in terms of their efficiency and scalability, this thesis proposes an iterative stochastic optimal control approach based on the generalized path integral formalism. More precisely, we suggest the use of the framework of stochastic optimal control with path integrals to derive a novel approach to RL with parameterized policies. While solidly grounded in value function estimation and optimal control based on the stochastic Hamilton Jacobi Bellman (HJB) equation, policy improvements can be transformed into an approximation problem of a path integral which has no open algorithmic parameters other than the exploration noise. The resulting algorithm can be conceived of as model-based, semi-model-based, or even model free, depending on how the learning problem is structured. The new algorithm, Policy Improvement with Path Integrals (PI2), demonstrates interesting similarities with previous RL research in the framework of probability matching and provides intuition why the slightly heuristically motivated probability matching approach can actually perform well. Applications to high dimensional robotic systems are presented for a variety of tasks that require optimal planning and gain scheduling.; In addition to the work on generalized path integral stochastic optimal control, in this thesis we extend model based iterative optimal control algorithms to the stochastic setting. More precisely we derive the Differential Dynamic Programming algorithm for stochastic systems with state and control multiplicative noise. Finally, in the last part of this thesis, model based iterative optimal control methods are applied to bio-mechanical models of the index finger with the goal to find the underlying tendon forces applied for the movements of, tapping and flexing. |
| Keyword | stochastic optimal control; reinforcement learning,; robotics |
| Language | English |
| Part of collection | University of Southern California dissertations and theses |
| Publisher (of the original version) | University of Southern California |
| Place of publication (of the original version) | Los Angeles, California |
| Publisher (of the digital version) | University of Southern California. Libraries |
| Provenance | Electronically uploaded by the author |
| Type | texts |
| Legacy record ID | usctheses-m3804 |
| Contributing entity | University of Southern California |
| Rights | Theodorou, Evangelos A. |
| Repository name | Libraries, University of Southern California |
| Repository address | Los Angeles, California |
| Repository email | cisadmin@lib.usc.edu |
| Filename | etd-Theodorou-4581 |
| Archival file | uscthesesreloadpub_Volume14/etd-Theodorou-4581.pdf |
Description
| Title | Page 153 |
| Contributing entity | University of Southern California |
| Repository email | cisadmin@lib.usc.edu |
| Full text | However, we will leave the discussion for PI2 for the next chapter and in this section we present the general version of iterative path integral control. In particular, we start by looking into the expectation (4.15) in the Feynman Kac Lemma that is evaluated over the trajectories " i = A xti , xti+1, ....., xtN B sampled with the forward propagation of uncontrolled diffusion dx = f (x, t)dt + B(x, t)Ldw. This sampling approach is inefficient since it is very likely that parts of the state space relevant to the optimal control task may not be reached by the sampled trajectories at once. In addition, it has poor scalability properties when applied to high dimensional robotic optimal control problems. Besides the reason of poor sampling, it is very common in robotics applications to have an initial controller-policy which is manually tuned and found based on experience. In such cases, the goal is to improve this initial policy by performing an iterative process. At every iteration (i) the policy *u(i−1) is applied to the dynamical system to generate state space trajectories which are going to be used for improving the current policy. The policy improvement results from the evaluation of the expectation (4.16) of the Feynman - Kac Lemma on the sampled trajectories and the use of the path integral control formalism to find *u(i). The old policy *u(i−1) is updated according to *u(i−1) + *u(i) and the process repeats again with the generation of the new state space trajectories according to the updated policy. In mathematical terms the iterative version of Path Integral Control is expressed as follows: V (i)(x) = min $u(i) J(x, u) = min $u(i) !" tN to ? q(x, t) + *u(i)T R *u(i) @ dt # (4.36) 139 |
