University of Southern California Dissertations and Theses

An improved iterative procedure for system identification with associative memory matrices

(USC Thesis Other)

An improved iterative procedure for system identification with associative memory matrices

PDF

Download

Open document

Flip pages

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content INFORMATION TO USERS
This manuscript has been reproduced from the microfilm master. UMI
films the text directly from the original or copy submitted. Thus, some
thesis and dissertation copies are in typewriter face, while others may be
from any type of computer printer.
The quality of this reproduction is dependent upon the quality of the
copy submitted. Broken or indistinct print, colored or poor quality
illustrations and photographs, print bleedthrough, substandard margins,
and improper alignment can adversely afreet reproduction.
In the unlikely event that the author did not send UMI a complete
manuscript and there are missing pages, these will be noted. Also, if
unauthorized copyright material had to be removed, a note will indicate
the deletion.
Oversize materials (e.g., maps, drawings, charts) are reproduced by
sectioning the original, beginning at the upper left-hand comer and
continuing from left to right in equal sections with small overlaps. Each
original is also photographed in one exposure and is included in reduced
form at the back of the book.
Photographs included in the original manuscript have been reproduced
xerographically in this copy. Higher quality 6 T x 9” black and white
photographic prints are available for any photographs or illustrations
appearing in this copy for an additional charge. Contact UMI directly to
order.
UMI
A Bell & Howell Information Company
300 North Zeeb Road, Ann Arbor MI 48106-1346 USA
313/761-4700 800/521-0600
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
An Improved Iterative Procedure for System Identification with Associative
Memory Matrices
by •
Daniel A. Poller
A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
Master of Arts
(Applied Mathematics)
December 1996
Copyright 1996 Daniel A. Poller
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UMI Number: 1383543
UMI Microform 1383543
Copyright 1997, by UMI Company. All rights reserved.
This microform edition is protected against unauthorized
copying under Title 17, United States Code.
UMI
300 North Zeeb Road
Ann Arbor, MI 48103
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
UNIVERSITY OF SOUTHERN CALIFORNIA
THE GRADUATE SCHOO L
UNIVERSITY PARK
LOE ANGELES. CALIFORNIA S 0 0 0 7
This thesis, written by
D w i l
under the direction of h~.15 Thesis Committee,
and approved by all its members, has been pre
sented to and accepted by the Dean of The
Graduate School, in partial fulfillment of the
requirements for the degree of
p afe Decem ber 1 7 , 1996
THESIS COMMITTEE
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Acknowledgments
Very few students are as fortunate as I to have an advisor as dedicated as Dr.
Proskurowski. His time and effort in the past months to my work has been enormous.
I thank him deeply for his dedication. I also thank Dr. Udwadia for his patience
and teaching. I thank Dr. Rosen for his logistical support. Of course the list is
much longer and I would like to express my gratitude to the many students of the
department of mathematics for their help and support during some storrnv moments.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Contents
Acknowledgm ents ii
List Of Tables iv
List Of Figures v
A bstract v
1 Introduction 1
2 A ssociative M em ory M atrices 4
2.1 Associative Memory A pproach.................................................................... - I
2.2 Initial Estimation of the Memory Matrix and Parameter vector . . . . 6
2.3 Iterative P ro ced u re....................................................................................... 7
2.4 The Algorithm ............................................................................................. 10
2.5 Associative Memory Matrices within Neural Networks............................ 12
3 Num erical Exam ples 14
3.1 Example 1 ....................................................................................................... 14
3.2 Example 2 ....................................................................................................... IS
3.3 Example 3 ....................................................................................................... 20
4 Conclusions 27
R eference List 28
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
List O f Tables
3.1 Results for the identification of the damped oscillator system. The
true parameter values are u > = 8.0, f = 0.09 and x0 = 20.0.-............... I.5
3.2 Estimated error vs. residual in 2-norm. Row (k) refers to the ktU
iteration.............................................................................................................. 16
3.3 Results for the identification of the damped oscillator system with
reduced damping. Column 1 gives the number of perturbed vectors
in each iteration................................................................................................ 17
3.4 Condition number of the memory matrix M vs. the number of iter
ations. Row 1 is the number of iterations, fc, and k is the condition
number of the memory matrix M ................................................................. 17
3.5 Condition number of the memory matrix M vs. the number of per
turbed vectors in each iteration. Row V refers to the number of
perturbed vectors in each iteration. Condition number k(.\[) was
computed at the 10f/l iteration...................................................................... 18
3.6 Identification of damped oscillator system in the presence of different
levels of RMS noise........................................................................................... 19
3.7 Effect of q on the results in presence of noise. Noise to signal ratio
was 5%................................................................................................................ 1 9
3.3 Results for the one parameter identification of the Van der Pol equation. 21
3.9 Results for the sirnultanious identification of a, :c0 and .f0 in the Van
der Pol equation................................................................................................ 21
3.10 Results for the identification of the parameter a in the Van der Poi
equation with different levels of RMS noise.................................................... 22
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
List Of Figures
3.L Response of the Damped O sc illato r............................................................
3.2 Response of the Damped Oscillator wit h Different Levels of RMS Noise
3.3 Response of the Van Der Pol Equation for different values of a . . .
3.4 Response of the Van Der Pol Equation with Different Levels of RMS
N oise...................................................................................................................
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Abstract
This paper presents significantly improved results for identification of parame
ters of mechanical systems using associative memory matrices. This was achieved
by modifying the iterative procedure of the previously developed method [l] and
[2]. Starting from an initial set of training vectors, this method generates the next
training vectors adaptively. The identification of the parameter vector is obtained
iteratively. Our numerical experiments show that the estimated parameter with the
presented method is more accurate by up to two orders of magnitude than with the
former algorithm. This is achieved with only a nominal increase in the computational
cost.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Chapter 1
Introduction
The concept of associative memory is quite old and has already been documented
in the writings of Aristotle [3]. For a biological memory to work it must have
some efficient method to store and to retrieve information. One of the ways this
is accomplished is by classifying the information by association. What has become
known as the classical laws of association between events are the following:
• I ) If the events occur simultaneously.
• 2) If they occur in close succession.
• 3) If they are similar.
• 4) If they are contrary.
I ’ o better understand these laws we need look only to our everyday lives. One may
try to recall the sequence of events when trying to-find a misplaced object, this is
related to the first two laws. As ah example of the third and fourth law we can
think of a good reproduction of the “Mona Lisa”. Although the reproduction is not
the actual painting, we are reminded of that famous work of art when viewing the
reproduction, giving us an example of the third law! If we are then given to view
a very bad reproduction of the “Mona Lisa”, we may be reminded of the former,
contrary, good reproduction.
At first the research in the area of associative memory was focused on achieving
a greater understanding of the human memory and how it functions. With the de
velopment of computers the focus of research in this area has changed and is now
1
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the implementation of associative memories in artificial intelligence.
Some of the early work in the field determined that the optimal associative linear
mapping, that is a mapping that associates two sets of vectors with each other, is the
generalized inverse [3]. This important fact will play a key role in the development of
our method. Most of the neural networks are examples of associative memory. The
idea behind a neural network is to associate inputs with outputs through a series
of connection weights. One of the special features of a neural network is that the
computations are both parallel and sequential rather than purely sequential. Some
of the general features of these networks are an input layer, an output layer and
possibly a number of hidden layers. The input layer receives the signal. The output
layer returns the result of the network. The hidden layers adjust the inputs with
corresponding weights to give the output. The weights are determined during an
initial training stage.
One of the early examples of a neural network is the Hopfield model [4]. In this
model there are a large number of nodes. Each node has two possible states:. +1
or -1. All the nodes of the network are connected. A different class of network are
the multi layer feed forward networks [5]. In this class of networks, all the nodes
in one layer are linked to all the others in the next layer. One.such network is
the multi layer percepton. Here the number of layers is specified in advance and
there are.no connection between the nodes of a given layer. One of the important
results is that a multi layer percepton with one hidden layer can approximate any
continuous function. Another network of the same class is the radial basis function
network. These networks have only one hidden layer and the output layer is merely
a linear combination of the hidden layer signals. Other developments ha,ve been the
use of recurrent neural networks such as the Elman network [6]. The idea behind
the recurrent networks is that the outputs of the system will be weighted and re
turned into the system as input. Other works in the field of associative memories are
the bidirectional associative memory [7], Kohonen networks and the learning vector
quantization network [5].
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Associative memories have been used in connection with various problems. Much of
the work has been done in the area of pattern recognition and self organizing maps
[3]. Another class of problems that have been addressed with associative memories
are system identification problems.
The system identification problem is the following: the signal and the model of the
system from which the signal was received are known. What is not known is the
parameters of the system [8]. As an example consider an oscillator with a connected
mass. It may be easy to measure the position and velocity of the mass over time.
Other facts about the system may be very difficult to determine such as damping
parameter of the system or the spring constant. The system identification problem
is to determine such facts about the system. Various methods have been developed
to identify the parameters of a system knowing the response of the system. Many
of such methods are variants of the Nev/ton-Gauss method [9]. Some methods are
probabilistic such as the extended Kalman filter. Extensive work has been done
in the field of neural networks, [5] and [6]. where both feedforward and reccurent
networks have been used. One of the difficulties with which most methods for sys
tem identification must deal is that it is necessary to have an. initial guess of the
parameter vector that is sufficiently close in value to the actual parameter vector.
A variant of the associative memory approach that successfully overcomes this dif
ficulty (at least in some instances) is that developed [b] and [21. In this paper we,
propose a modification of the above mentioned approach and in a series of numerical
experiments confirm its advantages: it gives more accurate estimates and also works
on previously intractable problems in the presence of noise.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Chapter 2
A ssociative M emory Matrices
2.1 A ssociative Memory Approach
One of the methods to perform an identification for a given system is to use an
associative memory approach. In this approach, the system is given input, then the
forward problem is solved with the given input. A response is then taken to be
the solutions of the forward problem. From this we can match up input response-
pairs. From these pairs a memory matrix is constructed which associates inputs
with the corresponding response. Here, the input-response pairs represent training.
The underlying idea behind this approach is that the memory matrix will have been
"trained” sufficiently so that it will acquire a knowledge base such that when we
expose the matrix to some new response it will then be able to accurately estimate
the input.
In this work the approach to the associative memories is based on that developed
by Kalaba and Udwadia [1] and [2]. In their work the parameters of the system are
directly taken to be the learning input. There are three advantages to this method:
(1) We always solve forward problems, which are usually easier then inverse prob
lems for the same system.
(2) The behavior of the forward problem is better understood then the inverse prob
lem.
(3) It is possible often to parameterize the initial conditions, thereby obtain esti
mates of both the input to the system as well as the parameters that describe the
system model.
A
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
This is a conceptual difference from past work in what constitutes input to the sys
tem for the purpose of training. This difference is believed to give their method
extremely good results.
The basis of the approach developed by Kalaba and Udwadia is as follows. Given a
parameter vector p ,- we can determine a corresponding response vector r t - by using
the given system. Hence for a set px, p2, . .. , pn we can calculate a set rq, r 2, . . . , rn.
Using these two sets we generate a memory matrix M. Then for an observed vector
r~ we estim ate the corresponding parameter vector p by letting p = Mr". Although
one would not expect a linear mapping to be able to estimate a non linear relation
ship between p the parameter vector and the response vector r, the results show
that this particular linear mapping often provides an accurate estimation of the pa
rameter vector.
An iterative procedure developed in [1] and [2] consists of the following steps: Using
an estimate of the parameter vector given by a memory matrix, generate a new
sequence of input vectors. Calculate the response from each new vector, update the
memory matrix separately for each new vector and calculate a new estimate for the
parameter vector. At the end of the procedure let the final approximation of the
parameter vector be the average of all the previous estimations. Using this method
the authors obtained unusually accurate results for the estimation of parameters in
systems that are well known to be hard to identify.
In this paper we propose a modification of the iterative procedure where instead
of updating the existing memory matrix, a new memory' matrix is constructed at,
each iteration. The crucial improvement is that here we will construct each new
training set to be in a closer neighborhood of the sought parameter vector than the
previous training set. This is done by changing certain weights of the procedure
at each iteration. We consider systems whose models are known in structure. The
parameter values of the system need to be identified. The dynamic inputs arc taken
to be known or to be described by parameters that may also need to be identified.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2.2 Initial Estimation of the Memory M atrix and
Param eter vector
In this section the description follows the presentation given in [ij and [2]. Consider
a system with a given response, r" and an unknown parameter vector. p* to be
identified. For this system, given any parameter vector, p,-, one can generate a
response vector, r,-. Let p: p, — > r< be a mapping from 5 R U to )RV. Let P' be a finite
set of parameter vectors and R! a finite set of response vectors. If for some p, and
Pj, i.j € /, r t - = rj, then we can remove r, from R' and pj from P ', receiving R and
P. Since the sets R and P are finite with the same number of vectors in each there
is an inverse p -1 : r t — > • pt. For a given p; and rt - we can.approximate p " 1 by a linear
mapping
P, = Mr,-. (2.1)
The linear mapping M is a u x u matrix that needs to be determined.
Theoretically we could look at the following matrix equation
P = M R (2.2)
and the solution to this equation is
M = PR+. (2.2)
where R+ is the pseudo inverse of R. M is the optimal associative linear mapping
between P and R [3]. In practice this result does not hold, ill conditioning of the
matrix M coupled with round off errors force us to modify Eq. 2.3.
To remedy this we construct a cost function, J , which associates the cost of limiting
0
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
the size of the elements of M with the cost of associating the set of parameter vectors
P with the set of response vectors, R.
J = oc\\MR — P\\2 + (1 — a )||M ||2, 0 < q < 1. (2 .4 )
If the norm is taken to be the Frobenious norm, ||M|| = \JTr (MTM) , by differen
tiating J with respect to M and setting the derivative to zero we obtain
M = P[ctRT(aR R T + (1 - a) A )-1]. (2.5)
When a = 1, M becomes the Generalized inverse solution (see Eq. 2.3), and the
m atrix M is sensitive to noise and round off error. As a decreases the sensitivity
decreases but so does the associativity between P and R.
2.3 Iterative Procedure
In this section we develop an iterative procedure that will allow us to compute
successive estimates, p of the parameter vector p“. Note that p and pp are vector
or vector components in the training sets while p and p,, are estimated vector or
vector components. Having obtained a memory matrix from equation 2.5 we
first compute an initial estimate, p'°* from a given response vector. r “, by setting
p (°) = M ( ( V . (2.6)
Having obtained an initial estimate p ^ for our parameter vector using the avail
able training set, we generate the first new training set of parameters, P^ K in some
neighborhood of p ^ , not necessarily a close one. The new set of n i training vectors
is generated by randomly perturbing, the estimated parameter vector. It is impor
tant to emphasize that the perturbations are done separately to each component of
the parameter vector, otherwise, all the newly perturbed vectors would be linearly
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
dependent. If p\j is the j th component, I < / < u, of the ith perturbed vector.
I < i < m in P (1) then
where 0 is a random number, — 0.5 < 9 < 0.5 and /?(l) is a weight indicating how far
we allow our perturbed vectors to be from the initial estimation.
Having a new training set, P^l\ we generate the corresponding response set,
The updated memory matrix, M^l\ is taken as
M l D = p W [ a R ( l )T (a R!-')R(l )T + (1 _ a ) I T ' ]
and a new estim ate of the parameter vector is given by
p W = ; V /( V
where r ' is the given response. The procedure in [1] and [2| uses each perturbed
vector separately to update the initial memory matrix. In our procedure v/e take
all the perturbed vectors and use them to create a new memory matrix without any
of the former vectors. The advantage of our procedure is that it uses only current
information that is hopefully closer to the real parameter vector than the initial data.
This process is then repeated, setting at the kth iteration
pW = p f - l)( l + 0 lk)B) (2.71
where 0 ^ is an updated weight, < 0^K ~1 ^ and 1 < i < r ? .f c . It may be useful
to take a different number of perturbed vectors in each iterations which is why we
subscript the parameter n-K . The memory matrix at each stage is
M(k) _ + (1 _ Q )/■)-•*] i.>.S)
and the estimated parameter is taken as
pW = Af(fc,r \ (2.9)
8
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
The idea behind changing the weights, f l k\ at each iteration is that at each iteration
the estimated vector, p ^ , is closer to the actual vector, p’ . The new training set
then will span a neighborhood that is closer to the actual vector. The updating of
the weight (3^ is the major improvement of our algorithm to the former works of
[1] and [2].
In real life applications one does not know the actual parameter vector p‘ to cal
culate the exact error, p* — p. Instead we can compute the residual which can be
viewed as an estimate of the error. This is done by using the estimated parameter
vector, ptk\ to solve the forward problem and obtain a response from the system.
r^kK Since \ve have the actual response of the system we can calculate the norm of
the residual as
This enables us to set a stopping criteria for the procedure in the following way:
we iterate until at the klh iteration ||r* — < r, where r is a given tolerance.
Then plfc l becomes the estimated parameter vector. This criteria may fail if we set
too stringent a tolerance, r. Concurrently, we set limit on the number of iterations.
itmax. Since the iterations rarely converge monotonically we choose as the estimated
parameter vector p the one chat satisfies
p = rnin ||r“ — 1 < k < itmax. (2.11)
This criterion allows to choose an optimal value of p even if H ? -" — r ~ (fc)|| diverges as
k increases, since we have a priori set a limit on the number of iterations.
9
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2.4 The Algorithm
Step 1 Obtain an estimate p using the training set S (which contains n parameter-
response pairs) by:
• a) For each p,, 1 < i < n, use the given system to solve for r,. Let
P = (Piip2 ---,Pn) and R = (rl; r2 . -. ,r„). Set S = (P ,P ),
• b) Compute the estimated memory matrix, M , using the training set . S’.
M = P{aR7{aR} R { 1 — a ) /) -1), where a is a constant. 0 < o < I.
• c) Compute the initial estimate of the parameter vector using the estimate of
the memory matrix p = Mr*.
Step 2 LIsing p, generate a new training set S 1 such that
Pi. 1 < i < «[, is a random perturbation of p. Form a new matrix and compute
• a) For 1 < i < rii and 1 < j < u, where n\ are the number of the ranclomh
generated perturbed vectors, u is the dimension of the parameter vector, let
p}j = pj( 1 + 0 ^ 9 ), where 0 ^ is a constant and 9 is a random number between
-0.5 and 0.5. f
• b) Using the given system compute r)L \ 1 Set, 5 (l' = (P ll‘. P (:').
• c) Compute a new estimated memory matrix,
= p M ( a R(VT(a RWTRP) + (1 _ o r ) / ) " 1.
• d) Calculate the new estimate of the parameter vector p(I* =
Step 3 For k — 2.3 continue iterative process such that at the k tfl iteration you
set:
1 0
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
• a) pj}} =plJ k~l)(l+ /3 (k> 0 ) and 0™ < 3 ^ ~ lK
• b) Using the system compute r[h\ L < i < tik- Set = {P^'K R ^ ) .
• c) Set the k ‘h estim ate of the memory matrix equal to
A f< *> = PW(aRWT{aRWTRW + (1 _
• d) Compute the kth estimated parameter vector p^k) =
until one of the stopping criteria is satisfied:
• a) j|r“ — r !fc)|| < r where t is a given tolerance and f i s the response obtained
by solving the forward problem with the estimated parameter vector at the kth
iteration,
• b) else having done itm ax iterations, itm ax set a priori, let the estimated
parameter vector, p ^ , be such that I satisfies mirii\\r" — 1 < / < itmax.
One should note that the stopping criteria a) was not used in the numerical ex
periments in Ch. 3 since it was difficult to guess a proper tolerance, r, for each
example.
v.
11
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
2.5 Associative M emory M atrices within Neural
Networks
Neural networks are a class of structures that associate a given input with an out
put. Since this association is achieved by having previously trained the network we
consider the neural network to have an associative memory. Many different types
of such neural networks exist but some of the basic charectaristics are a layered
structure of nodes and a learning-training method.
The two layers that are in almost every network are an input layer and an output
layer. There could be a varied number of inner layers to the network which are
referred to as hidden layers. Each layer consists of nodes that are connected within
and between the layers in different ways depending on the type of network. If the
inputs flow in only one direction we call the network a feedforward network. If some
of the outputs of nodes flow back to nodes in previous layers or into the same layer
we call the network a reccurent network Between the layers the values of the inputs
are changed by connection weights.
The determination of the connection weights is the task of the training of the net
work with some learning algorithm. If the the weights are adjusted during training
according to the known outputs of the system we say that the learning is supervised.
If only input patterns are shown to the network which then uses similarity of inputs
adapt the weights the learning is called unsupervised.
In this paper, the structure of the associative memory matrix we employ is that of
a rudimentary feedforward, supervised neural network. Recall that we are given a
response vector and are required to estimate the parameter vector corresponding
to that response. Assume the response, r*, to be a vector of dimension v and the
parameter vector, p* to be of dimension u. The input of the system consists of u
12
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
nodes where each node is given the response, r*. The inputs are weighted and the
result is transferred to the output layer.
In our approach, the training of the system is achieved through a proper construc
tion of the memory matrix. First the network is given both the input of the system
and the known output. Then a linear estimation of the mapping between input and
output is calculated (see Eq. 2.5). The addition of more training sets corresponds to
updating the weights. The multiplication of the ith row of the memory matrix with
the response, r* corresponds to the weighting of the r A input node (see Eq. 2.6).
The iterative procedure of the associative memory matrices we are using can be
viewed as the retraining of the network. That is to say we replace the connection
weights of the network by new weights are calculated using the previous output of
the network. This is performed by generating the new memory matrix. The new
connection weights correspond to the rows of the new memory matrix.
From this we can conclude that the associative memory matrix as presented here
is a subset of the class of neural networks. This may be helpful in the future to
interpret results with associative memory matrices by using known results from the
field of neural networks.
12
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Chapter 3
Numerical Examples
In this section we present numerical experiments that illustrate the efficiency of our
method. The maximal number of iterations, k. in each experiment was chosen a
priori to be itm ax = 15. The estimated parameter vector was taken at the kth
iteration where k satisfied Eq. (2-11). In the first two examples the optimal number
of iterations at which the estimated parameter vector was computed was kopt — 7
to 12 and in the third example ’ with ko p t between 2 and 4. The number of randomly
perturbed vectors generated at each iteration is n\ and is constant for all iterations
within one experiment. The weight is set at All the experiments
were performed using MATLAB.
3.1 Example 1
Consider a system described as a single degree-of-freedom damped oscillator gov
erned by the differential equation:
x [t) 4- 2u)£x(t) + o/2.r(t) = 0
x(t = 0) = .t0, i{t = 0) = T0
We consider the identification of the parameters ui , £ and x0 , starting with a set
of 5 training inputs. The true parameter vector p“ is taken to be [8,0.09,20]T. The
5 training vectors 1 < i < 5, were chosen as :
1-1
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
U ) %error
£
%error x'o %error
A 20.10 152 0.089 1.10 18.68 1.30
B 8.290 3.64 0.088 2.32 19.94 0.30
C 8.003 0.04 0.090 0.01 20.00 < 0.01
Table 3.L: Results for the identification of the damped oscillator system. The true
parameter values are u j = 8.0, £ = 0.09 and x0 = 20.0.
[6,0.02,10]T, [20,0.06,24]r , [16.0.03,1S]T, [10,0.01,25]r and [7,0.10, lo]T, the same
as in [1] . The mean of the training set is [11.8,0.044, lS.4]r and the standard de
viation is [6.02,0.036,6.27]T . Thus the training set is not in a close neighborhood
of the true parameter. One should take note that we are identifying simultaneously
two parameters of the system, w, f and the initial input into the system, x‘ 0.
The initial displacement x0 is taken to be zero. We use two windows of the response
data points; of them 10 points are taken from time zero equally spaced 0.02 units
apart and 10 points starting at 3.3 and equally spaced 0.3 units apart, see Pig. 3.1.
The parameter a was taken as 0.9999. In each iteration we used ni = 2 5 randomly
perturbed vectors. Row A are the results obtained with the initial training set with
out any iterations. Row B are the results obtained using the algorithm in [1]. Row
C are the results obtained using our procedure.
These results show a significant improvement over the results obtained using the al
gorithm described in [l]. There, the system and the training sets were identical, only
the value of a was taken to be a = 0.9 to ensure greater stability. Our algorithm
gives results that are more accurate of about two orders of magnitude than those
from [1]. This is especially important in the estimation of £. the damping parameter,
usually a difficult parameter to identify when the percent of critical damping is very
small.
It is important to relate the residual with the actual error. In our experiments the
actual parameter vector was known so it was possible to calculate the actual error
but this is not the case in general. In Table 3.2 we show the values of the residual,
at the k th iteration, ||r* — r ^ | | 2 and the values of the actual error of the estimated
15
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
(k)
1 2 5 10
H r*-rW IU 1.698 0.521 0.320 0.009
lip- - p1 *1 1,
1.121 0.270 0.158 0.004
Table 3.2: Estimated error vs. residual in 2-norm. Row (k) refers to the k th iteration.
parameter at the kth iteration, ||p* — p(k> ||2. These results refer to the experiment
described for the previous table. As the results show, we see a direct correlation
between the actual error and the residual. From these results we can see that the
use of the residual to determine the accuracy of our estimation of the param eter is
valid.
For the next experiment the damping parameter, £, was reduced from 0.09 to
0.005. This makes the damping parameter hard to estimate accurately. 1'sintr
only n\ = 25 vectors as in Row C of the previous experiment. Tabic 3.1.. the
error in £ increased from 0.01% to 29%. Nevertheless, by increasing t.he amount
of randomly perturbed vectors used in each iteration it is possible to get an ac
curate identification of this parameter as well, see Table 3.3 It should be noted
that the decrease in the errors is not strictly monotonic and that is due to the
random elements involved in running this process. The conditions, aside from
the damping parameter, were taken to be the same as in the experiment for Ta
ble la. The true parameter vector is [8.0,0.005,20.0]r . The training set was
[6,0.002,10]T, [20,0.001,24jr , [16,0.0l,lS]r , [10,0.001,25]r and [7,0.008, 15jr .
The computational cost connected with of increasing the number of vectors in each
iteration was negligible because the matrix operations in MATLAB are very efficient
and the system has an analytic solution.
It is not obvious how many iterations to perform or how many perturbed vectors
to create in each iteration. Some insight to this may be gained by analyzing the
memory matrix. /W , more closely. After a large number of iterations the accuracy
lti
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
1 u > %error
£
%error Xq %error
25 8.15 1.91 0.0064 29 19.34 3.30
30 8.01 0.12 0.0060 20 20.20 L.00
35 7.97 0.30 0.0088 76 20.04 0.20
40 8.00 0.01 0.0042 15 19.98 0.06
45 7.99 0.04 0.0059 19 19.98 0.07
50 8.00 0.04 0.0057 14 20.01 0.07
55 8.00 0.06 0.0044 10 19.98 0.05
60 8.00 0.05 0.0049 0.6 19.99 0.00
65 8.02 0.34 0.0061 23 20.02 0.11
70 8.00 < 0.01 0.0050 0.4 20.00 < 0.01
75 8.00 < 0.01 0.0050 1.4 20.00 0.03
Table 3.3: Results for the identification of the damped oscillator system with reduced
damping. Column 1 gives the number of perturbed vectors in each iteration.
1 0 1 5 10 15 20 | 30 50
*
103 429 230 299 1.21 0 6 3.5i09 | 4l09 2.5l012
Table 3.4: Condition number of the memory matrix M vs. the number of iterations.
Row 1 is the number of iterations, k, and k is the condition number of the memory
matrix M.
of the method will decrease due to an increase in the condition number of M. Tabie
3.4. This happens since, as (3^ decreases with each iteration, the elements of each
row in P become similar, causing M to be nearly singular (recall that M = PQ).
The conditions for Table 3.4. and Table 3.5. are identical to the experiment for Ta
ble 3.1, except for the number of iterations in Table 3.4. and number of perturbed
vectors in each iteration in Table 3.5
Table 3.5 illustrates the relationship between the number of perturbed vectors used
in each iteration and the condition number of the memory matrix M. As the num
ber of vectors increases the condition number is reduced since having more vectors
reduces the chance of entries in a given row of P being similar. As a resuit the
ill-conditioning of M is somewhat reduced.
17
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
V 20 25 30 40 50 75 100 200
K
4io9 3.51 09 1.91 0 9 1.31 0 9 1.21 0 9 5iqS bioS I.SiqS
Table 3.5: Condition number of the memory matrix M vs. the number of perturbed
vectors in each iteration. Row V refers to the number of perturbed vectors in each
iteration. Condition number k(M ) was computed at the 10t/l iteration.
3.2 Exam ple 2
In many applications the response signal will have some amount of noise. It is im
portant to see how the addition of noise affects the performance of our algorithm.
Table 3.6 presents the results of experiments after adding RMS noise to our signal
(see figure 3.2). Training vectors and integration region are identical with Table
3.1. The true parameter vector was taken as [8.00,0.09. 20.0]r . The results show
that l c and x0 can be identified quite accurately even in the presence of noise. The
relatively small magnitude of £ makes it; more sensitive to noise and a large increase
of the error is observed already for 1% RMS noise. The number of perturbed vectors
in each iteration had to be increased to 100 to obtain acceptable accuracy. We also
had to increase the stability of the method by reducing a to a = 0.94. In some of
the experiments conducted the estimated error increased significantly after a certain
number of iterations where in the experiment without noise. Table 3.1. the estimated
error decreased almost rnonotonically for all iterations.
We want to emphasize the importance of the value of a in the procedure. Recall.
a is the variable associating the cost function for the memory matrix, A/, (see Eq.
2.4) of optimally associating the training set with the response set and the size of
the elements of M. The accuracy of the algorithm in the experiment with noise is
18
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
noise U J %error
e
%error Xq %error
1% 8.20 0.2 0.068 23 19.56 2.1
2% 8.43 0.5 0.044 50 19.60 1.9
3% 8.00 0.1 0.058 34 19.56 2.1
4% 7.97 0.3 0.0661 26 19.52 2.4
5% 8.32 4.0 0.065 27 19.59 2.0
6% S.10 1.2 0.063 29 19.04 4.8
7% 8.25 3.2 0.051 42 19.41 2.9
8% 8.09 1.2 0.038 57 18.S6 5.9
9% 8.53 6.6 0.036 59 18.65 6.7
10% S. 54 6.7 0.040 55 19.37 3.1
Table 3.6: Identification of damped oscillator system in the presence cf different
Levels of RMS noise.
a U J %error
e
%error
i 0
%error
0.9999 7.00 12.4 -0.080 188 19.18 4.0
0.9900 8.31 3.90 0.052 41.0 19.10 4.4
0.9500 8.42 5.30 0.029 67.0 19.18 4.0
0.9000 8.09 1.10 0.075 16.0 19.35 3.2
0.8500 S. 24 3.00 0.048 45.0 19.37 3.1
Table 3.7: Effect of O ' on the results in presence of noise. Noise to signal ratio was
5%.
especially sensitive to a correct choice of a. In Table 3.7 we see that the size of a
is crucial for an accurate identification of the parameter vector in the presence of
noise. Each iteration had ni = 100 vectors.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
3.3 Exam ple 3
Consider the Autonomous Van der Pol equation given by
x — a(c — x 2)x + bx = 0
.-r(O ) = x0, ir(O) - xQ , t € [0,T],
where a.b and c are parameters. In the first set of experiments we assume that the
values of the parameters b and c are known, as well as xq and x0. In our experiment
the values are taken to be: x0 = 0.3, i 0 = — 0.3,6 = 5.0, c = 2.0. The unknown
parameter is the scalar a. To obtain the response vector, the forward equation was
solved using ODE45 routine with MATLAB. The tolerance of the routine was 10~'!.
(a Runge-Kutta-Fehlberg method). The cost of this procedure is higher than in the
previous examples because of the time consumed in the numerical solution of the
system. In particular, the cost is very high when the above values of the parameters
were chosen such that the system becomes highly non-linear. We integrated over the
interval [0,1], and for the response took the results of this integration at 20 points
at equidistant intervals. Only for the estimation of the parameter when the true
value was taken as 500. the integration was done over the interval [0, 0.02]. Each
new training set contained n i = 50 perturbed vectors. For a = 0.5 the mean of the
training set was 0.575 and the standard deviation 0.189, and scaled proportionally
for the other values of a taken. The parameter a was identified with extremely high
accuracy in very few (between k = 2 and k = 4) iterations even for large values of a
in which case the system is highly non linear (see Fig. 3.3). It should be noted that
although the estimation for the case with a = 500 was better than with a = 50. that
was due to the fact that the interval of integration was different.
We then increased the difficulty of the problem requiring that not only parameter a
but simultaneously also xti and xa be determined. The results are given in Table 3b
The five training vectors were: [1.0,0.1. — 0.35]r , [0.8, — 0.4, — 0.5]r . [0.2,0.5, — 0.35jr .
[0.3,0.6, — 0.4]r , [0.6, — 0.15. — 0.10]T (where [a,xo,io]T is the form of the vector).
The mean of this training set is [0.5S, 0.35, — 0.34]r and the standard deviation is
20
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
true value training set estimate % error
0.50 0.8,0.65,0.7,0.3,0.4,0.6 0.500 0.001
5.00 8,6.5,7,3,4,6 5.000 0.001
20.0 32,26,28,12,16,24 20.00 0.001
50.0 80,65,70,30,40,60 50.09 0.140
500 800,650,700,300,400,600 499.9 0.006
Table 3.8: Results for the one parameter identification of the Van der Pol equation.
a % error Xq % error
■1 . ■ -
Xo % error
A 0.475 5.00 0.312 4.14 -0.311 3.77
B 0.500 0.03 0.300 0.01 -0.300 0.004
Table 3.9: Results for the sirnultanious identification of a ,x o and .r0 in the Van der
Pol equation.
[0.33,0.22. — 0.15jr . The integration was taken over the interval [0,1] and the re
sponse was taken at twenty equidistant points. We set a = 0.999. At each iteration
r > i = 25 perturbed vectors were generated, the estimated parameter was taken after
k = 4 iterations. Row A are the results for this identification obtained in [2j. Row
B are the results obtained using the proposed method.
The results show a clear improvement of the results obtained in this work (row
B) from the results obtained in [2] (row A) by about two orders of magnitude.
In Table 3.10 we see that the effects of adding noise to the signal in the Van der
Pol equation is not significant (see figure 3.10). The true value of a = -5.0. We used
«i = 50 randomly perturbed vectors in each iteration.
The results achieved for the Van der Pol equation are surprising since we are using a
linear map to achieve extremely accurate identification of a non linear system. The
21
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
noise estimate % error
2% 4.999 0.01
4% 4.998 0.04
6% 5.008 0.16
8% 4.994 0.13
10% 4.988 0.23
Tabie 3.L0: Results for the identification of the parameter a in the Van der
equation with different levels of RMS noise.
results are robust in the sense that they are not sensitive to noise.
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
0 0.5 I 1.5 2 2.5 3 3.5 4.5
t im e
Figure 3.1: Response of the Damped Oscillator
23
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
0% Noise 1% Noise
3
2
0
• 2
0 0.5 1.5 i 2
time
5% Noise
o
u
0
•2
0.5 0 1.5 L
3
2
0
•2
0 0.5
time
fc Noise
3
2
0
•2
0 0.5
?
1.5
time time
Figure 3.2: Response of the Damped Oscillator with Different Levels of RMS Noise
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
a =0.5
a =5.0
1 0
5
0
■ 5
1 0
0 2 4 6 8 10
time
a= 20
X
-100
0 2 4 6 8 10
3 0
20
10
0
10
■ 2 0
■ 3 0
x 50
0 2 4 6 8 10
time
a =50
time
0 2 4 6 8 10
time
Figure 3.3: Response of the Van Der Pol Equation for different values of a
25
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
0% noise 1% noise
1 5
1 0
5
0
•5
0.8 0 0.2 0.4 0.6
time
5% noise
15
10
5
0
0 0.2 0.4 0.6 0.8 1
time
1 5
1 0
5
0
• 5
0 0.2 0.4 0.6 0.8 1
time
10% noise
15
10
5
0
•5
0 0.2 0.4 0.6 0.8 1
time
Figure 3.4: Response of the Van Der Pol Equation with Different Levels of RMS
Noise
26
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Chapter 4
Conclusions
This paper presents significantly improved results for the identification of parame
ters of mechanical systems using associative memory matrices. Our numerical ex
periments show that the estimated parameter vector with the presented method is
more accurate by up to two orders of magnitude than with the former algorithm,
see Ex.l and Ex.3. The systems that were identified were both linear and non linear
systems, with and without the presence of noise. The initial estimates of the param
eter vector were taken to be far from the true value. The improvement of the results
is achieved by modifying the iterative procedure of the associative memory method
for system identification presented in [1] and [2], without a significant increase in
computational cost. The modifications to the procedure were:
• The decrease of the weight 0 { k 1 at each iteration k.
• The generation of a new memory matrix, M at each iteration.
Future modifications of the procedure may include:
• 1) Changing a with each iteration for problems with noise. This would ensure
greater stability' of the method.
• 2) varying the number of randomly perturbed vectors, n*.. at each iterat ion to
help de.crease the condition number of the memory matrix, M.
• 3) Using a different weight 0 (k^ for each component of the perturbed vectors
instead of a common weight when good estimates are available for some of the
parameters that are to be identified.
27
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
Reference List
[1] R.E. Kalaba and F.E. Udwadia. An adaptive learning approach, to the identifica
tion of structural and mechanical systems. International Journal of Computers
and Mathematics with Applications, ‘ 22:67-75, 1990.
[2] R.E. Kalaba and F.E. Udwadia. Associative memory approach to the identifica
tion of structural and mechanical systems. Journal of Optimization Theory and
Applications, 76:207-223, 1993.
[3] T. Kohonen. Self-Organization and Associative Memory. Springer-Verlag, New
York, New York, 1988.
[4j J.J. Hopfieid and D.W Tank. Neural computations of decisions in optimization
problems. Biological Cybernetics, 52:141. 1985.
[5] K. Warwick G.W. Irwin and K.J. Hunt. Neural Networks Applications in Con
trol. The Institution of Electrical Engineers, London, United Kingdom. 1995.
[6] D.T. Pham and X. Liu. Neural Neiworks Applications for Identification. Predic
tion and Control. Springer-Verlag, London. United Kingdom. 1995.
[7] B. Kosko. Bidirectional associative memories. IEEE Transactions on Systems.
Man and Cybernetics, 18:49-60, 198S.
[8] L. Ljung. System Identification: Theory for the User. McGraw Hill, New York.
New York, 1988.
[9] R.E. Kalaba and K. Springarn. Control, Identification and Input Optimization.
Plenum, New York, New York, 1982.
2S
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
[10] T.L. Boullion and P.L. Odell. Generalized Inverse Matrices. Wiley-Interscience.
New York, New York, 1971.
[11] D.E. Catlin. Estimation, Control, and the Discrete Kalman Filter. Springer-
Verlag, New York, New York, 1989.
[12] S. Chen, S.A. Billings and P.M. Grant. Nonlineax system identification using
neural networks. International Journal of Control, 51:1191-1214. 1990.
[13] S. Chen, S.A. Billings. Neural networks for nonlinear dynamic system modelling
and identification. International Journal of Control, 56:319-346. 1992.
[14] Z. Hou and A. Lucifredi. On the use of neural networks for identification of
linear and nonlinear systems. Meccanica, 30:377-388, 1995.
[15] A.K. Jain, J. Mao and K.M. Mohiuddin. Artificial neural networks: a tutorial.
Computer, 29:31-44, 1996.
[16] H.H. Kagiwada, J.K. Kagiwada and S. Ueno. Kalaba's associative memories for
system identification. Appl. Math. Comput., 45:135-142, 1991.
[17] Y. Katnp and M. Hasler. Recursive neural networks for associative memory.
Wiley-Interscience, West Sussex, England, 1990.
[18] T. Kohonen. Self-Organizing Maps. Springer-Verlag, New York, New York.
1995.
[19] F. Peper and M.N. Shirazi. A categorizing associative memory using an adap
tive classifier and sparse coding. IEEE Transactions on Neural Networks, 7:669-675.
1996.
[20] R.M. Pringle and A.A. Rayner. Generalized Inverse Matrices. Charles Griffin
& Company, London. England, 1971.
[21] K.J. Raghunalh and V. Cherkassky. Noise performance of linear associative
memories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16:757-
765, 1994.
[22] 3. Zhang, A.G. Constantinides and L. Zou. Further noise rejection in linear
associative memories. Neural Networks, 5:163-168, 1992.
29
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A Fuzzy Controlled Nonuniform Diffusion Model For Competing Products

PDF

Efficient methods for nonlinear parabolic PDEs

PDF

A model reference adaptive control scheme for linear delay systems

PDF

Adaptive subdivision algorithm in postprocessing of compressed three-dimensional graphic data

PDF

Electricity demand as a function of temperature

$Computer simulation and control for production of an ideal diffraction pattern in optical fibers$

PDF

Computer simulation and control for production of an ideal diffraction pattern in optical fibers

PDF

Applications of the non-Gaussian stable process to communication engineering

PDF

Implementation aspects of Bézier surfaces to model complex geometric forms

PDF

Automated classification algorithms for high-dimensional data

PDF

General and explicit equations of motion for mechanical systems

PDF

Using The Cayley-Hamilton Theorem To Solve Linear Systems Of Difference Equations

PDF

Boundary element method for scattering of elastic waves in general anisotropic media

PDF

Investigation of soil-flexible foundation-structure interaction for incident plane SH waves

PDF

Investigation of a switching G mechanism for MEMS applications

PDF

Data driven control and identification: An unfalsification approach

PDF

Iterative detection for page -oriented optical data storage systems

PDF

Digital image processing for system identification

PDF

An energy method for seismic design

PDF

A global study of nonlinear dynamical systems by a combined numerical-analytical approach

PDF

Image stabilization using ridgelets

Asset Metadata

Creator Poller, Daniel Aviran (author)

Core Title An improved iterative procedure for system identification with associative memory matrices

Contributor Digitized by ProQuest (provenance)

School Graduate School

Degree Master of Arts

Degree Program Applied Mathematics

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag Computer Science,engineering, mechanical,Engineering, System Science,OAI-PMH Harvest

Language English

Advisor Proskurowski, Wlodek (committee chair), Rosen, Gary (committee member), Udwadia, Firdaus (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-9058

Unique identifier UC11337145

Identifier 1383543.pdf (filename),usctheses-c16-9058 (legacy record id)

Legacy Identifier 1383543.pdf

Dmrecord 9058

Document Type Thesis

Rights Poller, Daniel Aviran

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA