Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Recurrent neural networks with tunable activation functions to solve Sylvester equation
(USC Thesis Other)
Recurrent neural networks with tunable activation functions to solve Sylvester equation
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
UNIVERSITY OF SOUTHERN CALIFORNIA Recurrent neural networks with tunable activation functions to solve Sylvester equation by Siqi Zhang A thesis submitted in partial fulllment for the degree of Master of Science in the Department of Mathematics August 2017 Declaration of Authorship I, SIQI ZHANG, declare that this thesis titled, `Recurrent neural networks with tunable activation functions to solve Sylvester equation' and the work presented in it are my own. I conrm that: This work was done wholly or mainly while in candidature for a research degree at this University. Where any part of this thesis has previously been submitted for a degree or any other qualication at this University or any other institution, this has been clearly stated. Where I have consulted the published work of others, this is always clearly attributed. Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work. I have acknowledged all main sources of help. Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself. Signed: Date: i \The object of pure physics is the unfolding of the laws of the intelligible world; the object of pure mathematics that of unfolding the laws of human intelligence." James Joseph Sylvester ii UNIVERSITY OF SOUTHERN CALIFORNIA Abstract Department of Mathematics Master of Science by Siqi Zhang The Sylvester equation is a kind of important equations in control system, which has widely application in both engineering and science. Moreover, Recurrent Neu- ral Network has been proven is ecient way to solve nite-time stable problem. In this thesis we present a Recurrent Neural Network with tunable activation func- tion to improve the performance of existing Recurrent Neural Network. We would show it advantages theoretically through the rate of convergence. Furthermore, we would show it has better robustness by two practical examples. . . iii Acknowledgements I would rst like to thank my thesis advisor Associate Professor Neelesh V. Tiruvilu- amala of the Math department at University of Southern California. The door to Prof. Tiruviluamala oce was always open whenever I ran into a trouble spot or had a question about my research or writing. He consistently allowed this paper to be my own work, but steered me in the right the direction whenever he thought I needed it. I would also like to thank the members of the committee of my master thesis: Professor Robert J. Sacker and Professor Sergey V. Lototsky, who give me great pieces of advice to complete my thesis. Also, they are knowledgeable and patient mentor who give me plenty help when I am pursuing my master degree. Last but not least, I would like to thank my friends Mengxun Yan, Feng Ding, Yiliang Wang and Hao Liu, who give me much support in academics and daily life. Their accompany enrich my graduate school experience. iv Contents Declaration of Authorship i Abstract iii Acknowledgements iv List of Figures vii 1 Introduction 1 1.1 Introduction to Recurrent Neural Network(RNN) . . . . . . . . . . 1 1.2 Introduction to Finite-time stable problem . . . . . . . . . . . . . . 3 1.3 Introduction to Sylvester Equation . . . . . . . . . . . . . . . . . . 4 1.4 Structure of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Some classical methods to solve Sylvester equation 6 2.1 Roth's Removal Rule . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Bartels-Stewart Method . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Iterative Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3.1 Newton Iteration . . . . . . . . . . . . . . . . . . . . . . . . 11 2.3.2 The factorized iteration[2] . . . . . . . . . . . . . . . . . . . 11 3 RNN for Sylvester function 14 3.1 Background Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.2 Model Propose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.3 Analysis of nite-time convergence . . . . . . . . . . . . . . . . . . 22 3.4 Design of Zhang Neural Network(ZNN) . . . . . . . . . . . . . . . . 26 4 Experiment 28 4.1 Numerical Illustration . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.1.1 Rate of Convergence . . . . . . . . . . . . . . . . . . . . . . 30 4.1.2 Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 v Contents vi 4.2 Application: Solve Pseudo-inverse . . . . . . . . . . . . . . . . . . . 34 4.2.1 Rate of Convergence . . . . . . . . . . . . . . . . . . . . . . 35 4.2.2 Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5 Conclusion 39 Bibliography 40 List of Figures 3.1 Solution to equation 3.9 . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2 Solution to equation 3.10(r = 0:2, k 1 = 0:0001,k 2 = 15) . . . . . . . 19 4.1 Errors of example 4.1 with dierent activation function . . . . . . . 30 4.2 Solutions to example 4.1 with dierent activation function . . . . . 31 4.3 Solutions to example 4.1 with dierent activation function . . . . . 32 4.4 errors with noise of example 4.1 . . . . . . . . . . . . . . . . . . . . 33 4.5 Errors of example 4.2 with dierent activation function . . . . . . . 35 4.6 Solutions to example 4.1 with dierent activation function . . . . . 36 4.7 Solutions to example 4.1 with dierent activation function . . . . . 37 4.8 errors with noise of example 4.2 . . . . . . . . . . . . . . . . . . . . 38 vii I Dedicate this thesis to My Parents, Tingzhou Zhang and Xiaohong Lang. For giving me support Personally, Professionally and Spiritually and Their Love. viii Chapter 1 Introduction 1.1 Introduction to Recurrent Neural Network(RNN) Recurrent Neural Network is a class of articial neural network where connections between units form a directed cycle. Comparing to feedforward neural networks, RNNs can use their internal memory to process arbitrary sequences of inputs, which makes them applicable to tasks such as unsegmented connected handwrit- ing recognition[8] or speech recognition[25]. Also, due to its internal feedback, it shall also apply to the nonlinear dynamic system. To solve practical problem, Recurrent Neural Network is one of the most widely used methods in scientic and engineering elds, for example, pattern recognition[12] chaos system[17], op- timization problems and time-varying Sylvester equation[16]. In 1943, McCuloch and Pitts[20], come up with the formalized model of neurons. In the late 1940s psychologist Donald Hebb created a hypothesis of learning based on the mechanism of neural plasticity that is now known as Hebbian learning. 1 Introduction 2 Hebbian learning is considered to be a `standard' unsupervised learning rule, and its later variants were early models for long term potentiation[9]. In 1958, percep- tron, an algorithm for pattern recognition based on a two-layer computer learning network using simple addition and subtraction, is raised by Frank Rosenblatt[23]. Neural network research stagnated after the publication of machine learning re- search by Marvin Minsky and Seymour Papert[21] (1969), who discovered two fundamental issues with the computational machines that processed neural net- works. The rst was that basic perceptrons were incapable of processing the exclusive-or circuit. The second signicant issue was that computers didn't have enough processing power to handle the long run time required by large neural networks. Neural network research slowed until computers achieved greater pro- cessing power. In the 1960s, B. Widrow and M.Ho come up with ADALINE (Adaptive Linear Element)[28], which has been a power tool handling the adap- tive signal. A key advance that came later was the backpropagation algorithm, which was discovered by Werbos in 1974 [27]. It solved the exclusive-or problem, and more generally the problem of quickly training multi-layer neural networks. In the last two decades, Recurrent Neural network achieves rapid development. Recurrent Neural Networks may behave chaotically. In such cases, dynamical systems theory is useful for analysis. Recurrent neural networks are in fact recur- sive neural networks with a particular structure: that of a linear chain. Whereas recursive neural networks operate on any hierarchical structure, combining child representations into parent images, recurrent neural networks work on the linear progression of time, combining the previous time step and a hidden representation into the representation for the current time step. Introduction 3 1.2 Introduction to Finite-time stable problem Lyapunov stability is named after Aleksandr Lyapunov, a Russian mathematician who published his book The General Problem of Stability of Motion in 1892.[18] Lyapunov was the rst to consider the modications necessary in nonlinear systems to the linear theory of stability based on linearizing near a point of equilibrium. His work, initially published in Russian and then translated into French, received little attention for many years. Interest in it started suddenly during the Cold War period when the so-called \Second Method of Lyapunov" was found to ap- ply to the stability of aerospace guidance systems which typically contain strong nonlinearities not treatable by other methods. More recently the concept of the Lyapunov exponent (related to Lyapunov's First Method of discussing stability) has received broad interest in connection with chaos theory. Lyapunov stability methods have also been applied to nding equilibrium solutions in trac assign- ment problems.[26] It is known that stability of dynamic is always a focus of scholars. The nite- time stable system which is also called asymptotically stable system which could reach its equilibrium in nite time. Such kind of system has the higher rate of convergence, stronger robustness. In this article, we discuss the using recurrent neural network based on tunable activation function to solve the nite-time stable problem. Bhat and Bernstein built the foundation of nite-time stability and determined the necessary and sucient condition for nite-time stability.[3] More specically, Introduction 4 if and only if there exists Lyapunov function V such that _ V <lV (0<< 1), the system is nite-time stable. Sylvester equation is a widely applicable equation in engineering and science. In following chapters, we would discuss the nite-time stable problem of Sylvester function. 1.3 Introduction to Sylvester Equation Sylvester was a prolic 19th-century mathematician, the four-volume of his col- lected works totaling almost 3000 pages. He also led an eventful life, holding positions at ve academic institutions. For several years Sylvester was an actu- ary by day and did his mathematical research at night, and indeed he was one of the founders of the Institute of Actuaries. Surprisingly, he is the inventor of many currently used mathematical term. In 1850, he coined the term \matrix." Furthermore, he invented \minor" and \syzygy" in 1850, \canonical form," \dis- criminant" in 1851, and so on as credited in Oxford English Dictionary. [10] The Sylvester equation is the linear equation: AXXB =C; (1.1) where A2 R mm ;B2 R nn ;C2 R mn and X2 R mn is an unknown matrix. In 1884, Sylvester showed that equation 1.1 has a unique solution if and only if A and B has no eigenvalue in common. Classical Solution to Sylvester equation 5 In recent decades, Sylvester equations play a crucial role in applied mathematics and particularly in Control theory. Also, it has numerous application in signal processing, ltering, model reduction, image restoration, decoupling techniques for ordinary and partial dierential equations and block-diagonalization of matrices. 1.4 Structure of Thesis Chapter 1 is the introduction, here is the history, current state and primary ap- plications of Recurrent Neural Network, Finite-time stable problem and Sylvester equation. Moreover, in Chapter 2, it presents some classical methods to solve Sylvester equation.In Chapter 3, it is theoretically proved that the tunable activa- tion function has better performance than other functions. And then two example is present in Chapter 4 to show faster convergence and better robustness. Chapter 5 is the conclusion of this thesis. Chapter 2 Some classical methods to solve Sylvester equation 2.1 Roth's Removal Rule In 1952 W.E. Roth[24] showed such a theorem which has come to be known as Roth's Removal Rule: Theorem 2.1. Given equation 1.1, it has a solutionX if and only if the partitioned matrices D = 0 B @ A C 0 B 1 C A andD 0 = 0 B @ A 0 0 B 1 C A are similar. For ecient presentation of construction of the solution X in theorem, we need some techniques: 6 Classical Solution to Sylvester equation 7 i) The Kronecker product , dened for any pair (A;B) of matrices to be the partitioned matrix A B := 0 B B B B B @ a 11 B a 12 B a 21 B a 22 B 1 C C C C C A ii) The vec operator which applied to any matrix A gives the column vector obtained by stacking the columns of A in order: if A = (a 1 ; a 2 ; ),then vecA := 0 B B B B B @ a 1 a 2 1 C C C C C A So if A2 R mn ;B2 R pq then A B2 R mpnq and vecA2 R mn . Here is the steps to solve equation 1.1[7]: I. Construct the matrices P :=A I m+n I m D T 0 ; (2.1) Q :=C I m+n ; (2.2) S :=B I m+n I n D T 0 : (2.3) Let rankP =k and rankS =t. II. Choose any k linearly independent column from P, say p (1) ; p (2) ;:::; p (k) Classical Solution to Sylvester equation 8 III. Solve the linear equation k X i=1 i p (i) =Qvec 0 B @ O mn I n 1 C A where O mn is mn zero matrix, for k unknown numbers 1 ; 2 ;:::; k . IV. Obtain the matrix W from vec(W T ) = k X i=1 i e (i) ; say W = (W 1 ;W 2 ), where W2 R mm+n , W 1 2 R mm , W2 R mn V. Then X =W 2 is the solution. 2.2 Bartels-Stewart Method In 1972 Bartels and Stewart [1] published an article to show an algorithm to solve Sylvester equation 1.1: Classical Solution to Sylvester equation 9 I. matrixA andB are respectively reduced to lower and upper real Schur form A 0 and B 0 by othogonal similarity transformation U and V. A 0 =U T AU = 0 B B B B B B B B @ A 0 11 0 A 0 21 A 0 22 . . . . . . . . . A 0 m1 A 0 m2 A 0 mm 1 C C C C C C C C A (2.4) B 0 =V T BV = 0 B B B B B B B B @ B 0 11 B 0 12 B 0 1n B 0 22 B 0 2n . . . . . . 0 B 0 nn 1 C C C C C C C C A (2.5) where for all A 0 ii and B 0 ii , rankA 0 ii and rankB 0 ii are at most 2. II. Let C 0 =U T CV (2.6) and X 0 =U T XV (2.7) then Sylvester equation is equivalent to A 0 X 0 X 0 B =C 0 (2.8) III. Then equation 2.8 is equivalent to mn linear equations: A 0 kk X 0 kl +X 0 kl B ll =C 0 kl k1 X j=1 A 0 kj X 0 jl l1 X i=1 X 0 ki B il (2.9) For all k = 1; 2;:::;m and l = 1; 2;:::;n. Classical Solution to Sylvester equation 10 IV. Each equation is a linear system of order at most 4, sinceA 0 kk ;B 0 ll are of order at most 2. For example, if A 0 kk ;B 0 ll are of order 2, then: 2 6 6 6 6 6 6 6 6 4 a 0 11 +b 0 11 a 0 12 b 0 21 0 a 0 21 a 0 22 +a 0 11 0 b 0 21 b 0 21 0 a 0 11 +b 0 22 a 0 12 0 b 0 12 a 0 21 a 0 22 +b 0 22 3 7 7 7 7 7 7 7 7 5 2 6 6 6 6 6 6 6 6 4 x 0 11 x 0 21 x 0 12 x 0 22 3 7 7 7 7 7 7 7 7 5 = 2 6 6 6 6 6 6 6 6 4 d 11 d 21 d 12 d 22 3 7 7 7 7 7 7 7 7 5 (2.10) Where a 0 ij ;b 0 ij and x 0 ij denote the elements of A 0 kk ;B 0 ll ;X 0 kl , and d ij denote the elements of the right-hand side of equation 2.8. V. These equations can be solved for X 0 ij , for i = 1; 2;:::;m, j = 1; 2;:::;n. The solution of equation 1.1 is given by X =UX 0 V T . 2.3 Iterative Schemes The methods above are based on transforming the coecient to an easier to solve form and then solve the corresponding linear system of equations. Therefore, it is classied as direct methods. In this section we would present two methods based computing matrix sign function. Classical Solution to Sylvester equation 11 2.3.1 Newton Iteration Using the matrix sign function of D and theorem 2.1, Roberts [22] drives the following expression for the solution of the Sylvester equation 1.1: 1 2 (sign(D) +I m+n ) = 2 6 4 0 X 0 I 3 7 5 (2.11) It states that we can solve equation 1.1 by computing the matrix sign function of D. The most frequently used iterative scheme for computing the sign function is Newton iteration: Z 0 :=Z;Z k+1 := 1 2 (Z k +Z 1 k );k = 0; 1; 2;::: (2.12) And then he adapted it to the solution of Sylvester equation: A 0 :=A;A k+1 := 1 2 (A k +A 1 k ); (2.13) B 0 :=B;B k+1 := 1 2 (B k +B 1 k ); (2.14) C 0 :=C;C k+1 := 1 2 (C k +A 1 k C k B 1 k ): (2.15) for k = 0; 1; 2;:::. If we dene C 1 := lim k!1 C k , then X = 1 2 C 1 is the solution of equation 1.1. 2.3.2 The factorized iteration[2] To accelerate the computation, we reduce the dimension of matrices by factoriza- tion. That is:C = FG where F2 R mp ;G2 R pn and p m;n. And rewrite Classical Solution to Sylvester equation 12 the C k -iteration: F 0 :=F; F k+1 := F k ;A 1 k F k ; (2.16) G 0 :=G; G k+1 := 2 6 4 G k G k B 1 k 3 7 5 (2.17) C k+1 =F k+1 G k+1 : (2.18) Even though this iteration is cheap during initial steps, by doubling of columns of F k+1 and rows of G k+1 each step, it lose the advantage. This could be avoid by applying the technique introduced by Byers[4]: I. Suppose F k 2 R mp k ,G k 2 R p k n . Compute a rank-revealing QR factoriza- tion of G k+1 as dened in 2.17: 2 6 4 G k G k B 1 k 3 7 5 =URP;R = 2 6 4 R 1 0 3 7 5 ; (2.19) where U 2 R 2p k 2p k is orthogonal matrix, P 2 R nn is the permutation matrix and R2 R 2p k n is the upper triangular matrix with R 1 2 R rn of full row-rank. II. Compute a rank-revealing QR factorization of F k+1 U, with F k+1 dened in 2.16: F k ;A 1 k F k U =VTQ;T = 2 6 4 T 1 0 3 7 5 ; (2.20) where V 2 R mm is orthogonal matrix, Q2 R 2p k 2p k is the permutation matrix and T2 R m2p k is the upper triangular matrix with T 1 2 R t2p k of full row-rank. RNN for Sylvester function 13 III. Partition V = [V 1 ;V 2 ];V 1 2 R mt and compute [T 11 ;T 12 ] :=T 1 Q;T 11 2 R tr IV. Now we dene: F k+1 :=V 1 T 11 (2.21) G k+1 :=R 1 P (2.22) And it is easy to show C k+1 =F k+1 G k+1 (2.23) V. Set Y := lim k!1 F k Z := lim k!1 G k The solution to equation 1.1 is X =YZ. Chapter 3 RNN for Sylvester function In recent decades, in engineering and science, the recurrent neural network has achieved big success as an approach in pattern recognition[12] chaos system[17], optimization problems and time-varying Sylvester equation[16]. It is known that Hopeld Neural Network[11] could be used to solve the time-varying problem. In that case, applying recurrent neural network to resolve the time-varying problem becomes a hot research eld. Accordingly, lots of recurrent neural networks based on gradient are applicable to solve Sylvester equation[6][19][14]. However, in most of these articles, the norm of matrices is the primary judging method for net- work performance. The networks are designed to make norms converge to zero, but are short of the time-varying parameter to compensate speed. In this case, for time-varying problems norms could not converge to zero even in innite time. As a result, the networks lose their ecacy facing time-varying Sylvester equation. To solve time-varying problem, Yunong Zhang comes up with Zhang Neural Network[30] 14 RNN for Sylvester function 15 [29][31][32], which could converge to zero facing time-varying problem. Compar- ing to Bartel-Stewart algorithm [1], Zhang Neural Network provides the estima- tion error of asymptotic convergence and obtain approximate solution in nite steps.Moreover, Shuai Li comes[16] up with a new activation function, sign-bi- power: F (x) = 1 2 jxj r sign(x) + 1 2 jxj 1 r sign(x); 0<r< 1 (3.1) This function could accelerate convergence of Zhang Neural network. Obviously, whenkxk > 1, 1 2 jxj 1 r sign(x) plays crucial role to makekxk converges to 1 expo- nentially. On the other hand, whenkxk < 1, 1 2 jxj r sign(x) has decisive eect in makingkxk converge to 0 in nite time. To improve performance, we justify the sign-bi-power function by adding a linear term: F (x) = 1 2 jxj r sign(x) + 1 2 x + 1 2 jxj 1 r sign(x); 0<r< 1 (3.2) Evidently, the linear term 1 2 x enhance convergence rate wheneverkxk> 1 orkxk< 1. Furthermore, to further improve performance, we design a new tunable activation function k 1 ;k 2 ;k 3 which are greater than 0: F (x) = 1 2 k 1 jxj r sign(x) + 1 2 k 2 x + 1 2 k 3 jxj 1 r sign(x); 0<r< 1 (3.3) RNN for Sylvester function 16 3.1 Background Theory Considering such dynamic system: _ x =f(x(t));f(0) = 0;x2 R n ;x(0) =x 0 (3.4) where f : D! R n is a continuous function dened on an open neighborhood of origin x = 0. Denition 3.1. [3] The origin is said to be a nite-time-stable equilibrium of 3.4 if there exists an open neighborhood U D of the oringin and a function T : Unf0g! (0;1), called the settling-time function, such that the following statments holds: (i) Finite-time convergence: For everyx2 Unf0g, x is dened on [0;T (x)), x 2 Unf0g for all t2 [0;T (x)) and lim t!T (x) x (t) = 0. (ii) Lyapunov stability: For every open neighborhoodU of 0 there exists an open subsetU of U containing 0 such that, for every x2U nf0g, x (t)2U for all t2 [0;T (x)). The origin is said to be globally nite-time-stable equilibrium if it is a nite-time- stable equilibrium with D = U = R n . The following lemmas give the test condition for nite-time stability of system 3.4. Lemma 3.2. [3] If there exists a positive dened, dierentiable Lyapunov function V (x), real number k 1 > 0 and 0 < r < 1, in open neighborhood U R n , such that: _ V (x)j 3:4 k 1 V (x) r ;8x2 U (3.5) RNN for Sylvester function 17 Then the origin is nite-time-stable equilibrium and the convergence time T 1 (x 0 ) depending on x 0 satises: T 1 (x 0 ) V (x 0 ) 1r k 1 (1r) ;x 0 2 U: (3.6) Moreover, if U = R n and V (x) is radially unbounded (that is, when kxk ! +1;V (x)! +1), the origin is globally nite-time-stable equilibrium. Lemma 3.3. [13] If there exists a positive dened, dierentiable Lyapunov func- tion V (x)2 C 1 , real number k 1 ;k 2 0, k 1 +k 2 > 0 and 0 < r < 1, in open neighborhood U R n , such that: _ V (x)j 3:4 k 1 V (x) r k 2 V (x);8x2 U: (3.7) Then the origin is nite-time-stable equilibrium and the convergence time T 2 (x 0 ) depending on x 0 satises: T 2 (x 0 ) ln (1 + k 1 k 2 V (x 0 ) 1r ) k 2 (1r) ;x2 U (3.8) Moreover, if U = R n and V (x) is radially unbounded (that is, when kxk ! +1;V (x)! +1), the origin is globally nite-time-stable equilibrium. According to lemma 3.2 and 3.3, we know the upper bound of convergence time is depending on r and the less r is, the time is shorter. But when r goes to 0,jxj r is approximately equal to sign(x) which occurs vibration. Example 3.1. Consider such a dierential equation: _ x(t) =jx(t)j r sign(x(t)); 0<r< 1: (3.9) RNN for Sylvester function 18 When choosing dierent r's, the curve of solutions to equation 3.9 shows in g- ure3.1. From gure 3.1, it is easy to know when r =0.2, the convergence speed is (a) curves for dierent r's (b) enlarged view of curve for r=0.2 while x is close to zero Figure 3.1: Solution to equation 3.9 fastest but strong vibration also exists. To overcome vibration we choose a small k 1 and a large k 2 , so it is such equation: _ x(t) =k 1 jx(t)j r sign(x(t))k 2 x(t); 0<r< 1: (3.10) Figure 3.2 shows curve of equation 3.10. In the illustration, it shows the vibration is gone. In sliding mode control(SMC) the method with addingk 2 x is call Reaching Control.[1] Similar to the lemma 3.2 and 3.3, we try to get the convergence time of system3.4 in new condition which will lead to the convergence time of ZNN with tunable sign-by-power, which we will discuss in next section. RNN for Sylvester function 19 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 (a) curves for solution 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 -0.01 -0.005 0 0.005 0.01 0.015 0.02 0.025 (b) enlarged view of curve for r=0.2 while x is close to zero Figure 3.2: Solution to equation 3.10(r = 0:2, k 1 = 0:0001,k 2 = 15) Lemma 3.4. If there exists positive dened dierentiable Lyapunov function, in open neighorhood U R n , such that: _ V (x)j 3:4 k 1 V (x) r k 2 V (x)k 3 V (x) 1 r ;8x2 U (3.11) wherek 1 ;k 2 ;k 3 > 0 are tunable, and 0<r< 1, then the origin is nite-time-stable equilibrium and the convergence time T 3 (x 0 ) depending on x 0 satises: T 3 (x 0 ) 8 > < > : r ln [ k 1 +k 2 k 2 V(x 0 ) (r1)=r+k 3 ] k 2 (1r) + ln (1+ k 1 k 2 ) k 2 (1r) ; V (x 0 ) 1;8x2 U; ln [1+ k 2 k 1 V (x 0 ) 1r ] k 2 (1r) ; V (x 0 )< 1;8x2 U: (3.12) Moreover, if U = R n and V (x) is radially unbounded (that is, when kxk ! +1;V (x)! +1), the origin is globally nite-time-stable equilibrium. Proof. According to lemma 3.2, the nite-time stability is obvious. In that case, we need only to prove equation 3.12. Part one : If V (x 0 )> 1, RNN for Sylvester function 20 According to equation 3.11, _ V (x)k 2 V (x)k 3 V (x) 1=r ; (3.13) Multiply by integrating factor e k 2 t , d(e k 2 t V (x)) (e k 2 t V (x)) 1=r k 3 e 1 1 r k 2 t dt: (3.14) Integrate from 0 to t on both side, V (x)e k 2 t [V (x 0 ) 1 1 r + k 3 k 2 k 3 k 2 e (1 1 r )k 2 t ] r r1 (3.15) Make the left hand be equal to 1, t 1 = r ln [ k 2 +k 3 k 2 V (x 0 ) (r1)=r +k 3 ] k 2 (1r) ; (3.16) Now: when tt 1 , V (x) 1. Then, according to equation 3.11, _ V (x)k 1 V (x) r k 2 V (x): (3.17) By lemma 3.3 t 2 = ln 1 + k 2 k 1 k 2 (1r) (3.18) Then when t t 2 , x(t) = 0, when V (x 0 ) > 1 and t > t 1 +t 2 , x(t) = 0. That is, the rst inequation of equation 3.12 is proven. Part two: IfV (x 0 ) 1, inequation 3.17 is satised. According to lemma 3.3, when tT 2 (x 0 ), x(t) = 0 RNN for Sylvester function 21 Remark 3.5. Now, compare upper bounds o T 1 (3.6), T 2 (3.8) and T 3 (3.12). Obviously,8x 0 2 U, ln [1 + k 2 k 1 V (x 0 ) 1r ] k 2 (1r) < V (x 0 ) 1r k 1 (1r) , that is T 2 <T 1 ; For T 2 and T 3 : When V (x 0 )< 1, by lemma 3.3 and lemma 3.3, T 2 =T 3 . Meanwhile, when V (x 0 ) 1, ln [1+ k 2 k 1 V (x 0 ) 1r ] k 2 (1r) r ln [ k 2 +k 3 k 2 V(x 0 ) (r1)=r +k 3 ] k 2 (1r) ln (1 + k 2 k 1 ) = 1 k 2 (1r) ln (1+ k 2 k 1 V (x 0 ) 1r )(k 2 +k 3 V (x 0 ) 1r r ) r (1+ k 2 k 1 )(k 2 V (x 0 ) 1r r +k 3 V (x 0 ) 1r r ) r (3.19) As a result, when V (x 0 ) is large enough, T 2 is much larger than T 3 . 3.2 Model Propose Consider following time-varying Sylvester equation: A(t)X(t)X(t)B(t) +C(t) = 0: (3.20) Where t is time, A(t);B(t);C(t) are known time-varying matrices with a suitable number of dimension. X(t) is the unknown quantity. As a result, the target is to obtain the time-varying solution X(t). RNN for Sylvester function 22 Then the Zhang Neural Network[30] [29][31] to solve system 3.20 is: A(t) _ X(t) _ X(t)B(t) = _ A(t)X(t) +X(t) _ B(t) _ C(t) oF (A(t)X(t)X(t)B(t) +C(t)) (3.21) Where o2 R; o> 0, F (:) is activation function. It is expounded by Yunong Zhang[30] [29][31] that activation function has a huge eect on the speed of convergence. In spite of the simplicity of activationF (x) =x, it reduces the rate of convergence signicantly, so that it might spend an innite time to solve Sylvester equation accurately. It is already proven that sign-by- power activation function [16],that is 3.1, could accelerate convergence so that it could converge in nite time. However, it neglects 1 2 jxj 1=r sign(x) when estimating convergence time. To improve the performance, we come up with two new activation functions. One is to add a linear term x 2 in sign-by-power function, that is F 1 (x)3.2. The other one is to add tunable parameter on activation function 3.2, that is F 2 (x)3.3. We will discuss the nite-time stability thoroughly in following section. 3.3 Analysis of nite-time convergence In this section, stability of Zhang Neural Network with activation functionF 1 (x)3.2 and F 2 (x)3.3 will be discussed respectively. RNN for Sylvester function 23 In the rst place, using activation functionF 1 (x)3.2, following theorem is satised: Theorem 3.6. Using activation function F 1 (x) 3.2, for any initial value X 0 , Zhang neural network 3.21could converge to the theoretical solution of Sylvester equation 3.20, X (t). Moreover the convergence time T 4 is satised to: T 4 8 > < > : 2r ln [ 2 V(0) (r1)=2r +1 ] (1r) + 2 ln 2 (1r) ; V (0) 1; 2 ln [1+V (0) (1r)=2 ] (1r) ; V (0)< 1: (3.22) Where V (0) = (e + (0)) 2 ,e + (0) is the initial value of e + (t), 2 R;> 0and V (t) is Lyapunov function: V (t) =je + (t)j 2 : (3.23) Proof. Let error matrix: E(t) =A(t)X(t)X(t)B(t) +C(t): (3.24) Then we could get the dierential equation : _ E(t) = _ A(t)X(t) +A(t) _ X(t) _ X(t)B(t)X(t) _ B(t) + _ C(t) (3.25) Then by Zhang neural network 3.21, _ E(t) = oF 1 (E(t)); (3.26) _ e ij (t) =F 1 (e) ij (t)): (3.27) Where e ij (t)2 R is the entry in i-th row and j-th column of matrix E(t). RNN for Sylvester function 24 By equation 3.27, the elements of error matrix are mutually independent. Let e + (0) = maxje ij (0)j, then every element has independent dierential equation _ e ij (t) =F 1 (e ij (t)). and je + (t)je ij (t)je + (t)j;t 0;8i;j: (3.28) So that, when e + (t) converges to 0,8i;j, e ij (t) converges to 0. Then we have: _ e + (t) =F 1 (e + (t));e + (0) =maxje ij (0)j: (3.29) Then derivative 3.23, _ V (t) = 2e + (t)F 1 (e + (t)) = (je + (t)j r+1 +je + (t)j 2 +je + (t)j 1=r+1 ) = (V 1+r 2 +V +V r+1 2r ): (3.30) When V (0) =je + (0)j 2 1, by lemma 3.4, T 4 2 ln [1 +V (0) (1r)=2 ] (1r) (3.31) thus when tT 4 ,je + (t)j = 0. When V (0) =je + (0)j 2 1, by lemma 3.4, 2r ln [ 2 V (0) (r1)=2r +1 ] (1r) + 2 ln 2 (1r) (3.32) thus when tT 4 ,je + (t)j = 0. Then, using activation function F 1 (x)3.2, following theorem is satised: RNN for Sylvester function 25 Theorem 3.7. Using tunable activation function F 2 (x) 3.3, for any initial value X 0 , Zhang neural network 3.21 could converge to the theoretical solution of Sylvester equation 3.20, X (t). Moreover the convergence time T 5 is satised to: T 4 8 > < > : 2r ln [ k 2 +k 3 k 2 V(0) (r1)=2r +k 3 ] k 2 (1r) + 2 ln 1+ k 2 k 1 k 2 (1r) ; V (0) 1; 2 ln [1+ k 1 k 2 V (0) (1r)=2 ] k 2 (1r) ; V (0)< 1: (3.33) Where 0<r < 1;k 1 ;k 2 ;k 3 are tunable positive parameter, V (0) = (e + (0)) 2 ,e + (0) is the initial value of e + (t) and 2 R;> 0 Proof. By the proof of theorem 3.6, _ E(t) = oF 1 (E(t)); (3.34) _ e ij (t) =F 1 (e) ij (t)): (3.35) Then we have: je + (t)je ij (t)je + (t)j;t 0;8i;j: (3.36) Thus, when e + t converges to 0,8i;j, e ij (t) converges to 0. Thus, _ e + (t) =F 2 (e + (t));e + (0) =maxje ij (0)j: (3.37) Then derivative 3.23, _ V (t) = 2e + (t)F 2 (e + (t)) = (k 1 je + (t)j r+1 +k 2 je + (t)j 2 +k 3 je + (t)j 1=r+1 ) = (k 1 V 1+r 2 +k 2 V +k 3 V r+1 2r ): (3.38) RNN for Sylvester function 26 Similary, when V (0) =je + (0)j 2 1, by lemma 3.4, T 5 2r ln [ k 2 +k 3 k 2 V (0) (r1)=2r +k 3 ] k 2 (1r) + 2 ln 1 + k 2 k 1 k 2 (1r) (3.39) thus when tT 5 ,je + (t)j = 0. When V (0) =je + (0)j 2 1, by lemma 3.4, T 5 2 ln [1 + k 1 k 2 V (0) (1r)=2 ] k 2 (1r) (3.40) thus when tT 5 ,je + (t)j = 0. 3.4 Design of Zhang Neural Network(ZNN) According to theorem 3.6, 3.7 and Theorem 2 in article [16]. Given initial value e + (0) and upper bound of convergence time T b , we could design the parameter of ZNN. For example, by Theorem 2 in article [16], the parameter is satised to: 2je + (0)j 1r (1r)T b : (3.41) Note that the actual convergence time of ZNN based on sign-by-power activation function T a is less than T b . Suppose r is know, T a will decrease with the increase of . Hence, the dierence between T b and T a will increase. By inequation 3.41, let = 2je + (0)j ( 1r) (1r)T b Experiment 27 AS a result, parameter and r could be determined by following: T b = 2je + (0)j ( 1r) (1r) (3.42) By theorem 3.7, ;r;k 1 ;k 2 ;k 3 is statised to following: T b 8 > < > : 2r ln [ k 2 +k 3 k 2 V(0) (r1)=2r +k 3 ] k 2 (1r) + 2 ln 1+ k 2 k 1 k 2 (1r) ; (e + (0)) 2 1; 2 ln [1+ k 1 k 2 V (0) (1r)=2 ] k 2 (1r) ; (e + (0)) 2 < 1: (3.43) Similarly, we could choose parameter ;r;k 1 ;k 2 ;k 3 such that : T b = 8 > < > : 2r ln [ k 2 +k 3 k 2 V(0) (r1)=2r +k 3 ] k 2 (1r) + 2 ln 1+ k 2 k 1 k 2 (1r) ; (e + (0)) 2 1; 2 ln [1+ k 1 k 2 V (0) (1r)=2 ] k 2 (1r) ; (e + (0)) 2 < 1: (3.44) Chapter 4 Experiment In this chapter, a numerical illustration 4.1 and an application example 4.2 will be given to compare the performance of dierent activation functions: 1. power-by-sign activation function F (x)3.1: F (x) = 1 2 jxj r sign(x) + 1 2 jxj 1 r sign(x); 0<r< 1 (4.1) 2. power-by-sign activation function with a linear term F 1 (x)3.2 F 1 (x) = 1 2 jxj r sign(x) + 1 2 x + 1 2 jxj 1 r sign(x); 0<r< 1 (4.2) 3. tunable power-by-sign activation function with a linear term F 2 (x)3.3 F 2 (x) = 1 2 k 1 jxj r sign(x) + 1 2 k 2 x + 1 2 k 3 jxj 1 r sign(x); 0<r< 1 (4.3) 28 Experiment 29 4. linear activation function F l (x) F l (x) =x (4.4) 5. bipolar-sigmoid activation functionF s (x) [31] F s (x) = 1 +e r 2 1e r 2 1e r 2 x 1 +e r 2 x ;r 2 > 2 (4.5) 6. power-sigmoid activation function F ps (x)[15] F ps (x) = 1 2 x r 1 + 1 2 1 +e r 2 1e r 2 1e r 2 x 1 +e r 2 x ;r 1 3;r 2 > 2 (4.6) 4.1 Numerical Illustration Example 4.1. For Sylvester equation 3.20, A(t) = 2 6 4 sin(t) cos(t) cos(t) sin(t) 3 7 5 ;B(t) = 2 6 4 0:1sin(t) 0 0 0:2cos(t) 3 7 5 ,C(t) = 2 6 4 0:1sin 2 (t) 1 0:2cos 2 (t) 0:1sin(t)cos(t) 0:2sin(t)cos(t) 1 3 7 5 . We know the theoretical solution is X (t) = 2 6 4 sin(t) cos(t) cos(t) sin(t) 3 7 5 . Experiment 30 4.1.1 Rate of Convergence Figure 4.2 and 4.3 show curves using dierent activation function. They all con- vergence before t = 10 but convergence time varies. Moreover, Figure 4.1 shows estimation error withkX(t)X (t)k F , which could result into that the tunable sigh-by-power activaton function with suitable parameter converges fasteast com- paring to other functions. 0 1 2 3 4 5 6 7 8 9 10 0 0.5 1 1.5 2 2.5 3 3.5 4 sign-by-power with linear term tunable sign-by-power sign-by-power bipolar sigmoid power sigmoid linear Figure 4.1: Errors of example 4.1 with dierent activation function Experiment 31 (a) sign-by-power function with r = 0:5 (b) sign-by-power with linear term function with r = 0:5 (c) tunable sign-by-power with r = 0:5;k 1 =k 3 = 1;k 2 = 10 Figure 4.2: Solutions to example 4.1 with dierent activation function Experiment 32 (a) linear activation function (b) bipolar sigmoid function with r = 2 (c) power sigmoid function with r = 3;p = 3 Figure 4.3: Solutions to example 4.1 with dierent activation function Experiment 33 4.1.2 Robustness In real application, it is inescapable to be disrupted by noise. Higher robustness to avoid noise is also one of target. Adding extra white noise v with =0:1into ZNN 3.21: A(t) _ X(t) _ X(t)B(t) = _ A(t)X(t) +X(t) _ B(t) _ C(t) oF (A(t)X(t)X(t)B(t) +C(t)) +v (4.7) In gure 4.4, it is shown that ZNN could converge to approximate solution but the performance varies with dierent activation function. Also, it is a reasonable hypothesis that tunable sign-by-power activation function with suitable parameter has higher robustness than the others. 0 1 2 3 4 5 6 7 8 9 10 0 0.5 1 1.5 2 2.5 tunable sign-by-power sign-by-power bipolar sigmoid Figure 4.4: errors with noise of example 4.1 Experiment 34 4.2 Application: Solve Pseudo-inverse Pseudo-inverse of matrix is widely used in dierent elds, such as Least Square Method. For any time-varying matrix J(t)2 R mn , its pseudo-inverse J (t)is usually dened as [5] J (t) = (J T (t)J(t)) 1 J T (t)(mn) and J (t) =J T (t)(J T (t)J(t)) 1 (m<n) . For Sylvester equation 3.20, let A(t) =J T (t)J(t);B(t) = 0;C(t) =J T (t)(mn) or A(t) = 0;B(t) =J(t)J T (t);C(t) =J T (t)(m<n) Then the solution X(t) is the pseudo-inverse J (t). In following example, we would compare the performance among dierent activation function by rate of convergence and robustness. Example 4.2. Solve the pseudo-inverseJ (t) ofJ(t) = 2 6 4 0:5cos(t) sin(t) 0:5cos(t) 0:5sin(t) cos(t) 0:5sin(t) 3 7 5 . It is easy to know the theoretical solution J (t) = 2 6 6 6 6 6 4 cos(t) sin(t) sin(t) cos(t) cos(t) sin(t) 3 7 7 7 7 7 5 . Experiment 35 Since m < n, let A(t) = 0;B(t) = J(t)J T (t) = 2 6 4 3 4 cos(2t) 4 sin(2t) 4 sin(2t) 4 3 4 + cos(2t) 4 3 7 5 ;C(t) = J T (t) = 2 6 6 6 6 6 4 0:5cos(t) 0:5sin(t) sin(t) cos(t) 0:5cos(t) 0:5sin(t) 3 7 7 7 7 7 5 . 4.2.1 Rate of Convergence Figure 4.6 and 4.7 show curves using dierent activation function. They all con- vergence before t = 10 but convergence time varies. Moreover, Figure 4.5 shows estimation error withkX(t)J (t)k F , which could result into that the tunable sigh- by-power activaton function with suitable parameter converges fasteast comparing to other functions. 0 1 2 3 4 5 6 7 8 9 10 0 0.5 1 1.5 2 2.5 3 3.5 4 sign-by-power with linear term tunable sign-by-power sign-by-power sigmoid power sigmoid linear Figure 4.5: Errors of example 4.2 with dierent activation function Experiment 36 (a) sign-by-power function with r = 0:5 (b) sign-by-power with linear term function with r = 0:5 (c) tunable sign-by-power with r = 0:5;k 1 =k 3 = 1;k 2 = 10 Figure 4.6: Solutions to example 4.1 with dierent activation function Experiment 37 (a) linear activation function (b) bipolar sigmoid function with r = 2 (c) power sigmoid function with r = 3;p = 3 Figure 4.7: Solutions to example 4.1 with dierent activation function Conclusion 38 4.2.2 Robustness Similarly, adding a noise into ZNN 3.21, we have 4.7. In gure 4.8, it is shown that ZNN could converge to approximate solution but the performance varies with dierent activation function. Also, it is a reasonable hypothesis that tunable sign- by-power activation function with suitable parameter has higher robustness than the others. 0 1 2 3 4 5 6 7 8 9 10 0 0.5 1 1.5 2 2.5 bipolar sigmoid sign-by-power tunable sign-by-power Figure 4.8: errors with noise of example 4.2 Chapter 5 Conclusion In this paper, we rst introduce some classical methods to solve Sylvester equa- tion. But most of them could not be applied to solving time-varying Sylvester function. Then we present the Zhang Neural Network to address time-varying Sylvester function. Based on the nite-time stable theory we design a tunable activation function to improve the performance of the ZNN, comparing to some classical activation functions such as linear activation function, bipolar sigmoid function, power-sigmoid function, and sign-by-power function. Then we show the tunable activation function with appropriate parameter has faster convergence rate and stronger robustness comparing to the activation func- tion mentioned above. 39 Bibliography [1] R. H. Bartels and G. Stewart, Solution of the matrix equation ax+ xb= c [f4], Communications of the ACM, 15 (1972), pp. 820{826. [2] P. Benner, Factorized solution of sylvester equations with applications in control, sign (H), 1 (2004), p. 2. [3] S. P. Bhat and D. S. Bernstein, Finite-time stability of continuous au- tonomous systems, SIAM Journal on Control and Optimization, 38 (2000), pp. 751{766. [4] R. Byers, Solving the algebraic riccati equation with the matrix sign function, Linear Algebra and its Applications, 85 (1987), pp. 267{279. [5] S. L. Campbell and C. D. Meyer, Generalized inverses of linear trans- formations, SIAM, 2009. [6] F. Ding and T. Chen, Gradient based iterative algorithms for solving a class of matrix equations, IEEE Transactions on Automatic Control, 50 (2005), pp. 1216{1221. [7] F. Gerrish and A. Ward, Sylvester's matrix equation and roth's removal rule, The Mathematical Gazette, 82 (1998), pp. 423{430. 40 Bibliography 41 [8] A. Graves, M. Liwicki, S. Fern andez, R. Bertolami, H. Bunke, and J. Schmidhuber, A novel connectionist system for unconstrained hand- writing recognition, IEEE transactions on pattern analysis and machine intel- ligence, 31 (2009), pp. 855{868. [9] D. Hebb, Organization of behavior. new york: Science editions, 1961. [10] N. J. Higham, Sylvester's in uence on applied mathematics, (2014). [11] J. J. Hopfield, Neural networks and physical systems with emergent collec- tive computational abilities, Proceedings of the national academy of sciences, 79 (1982), pp. 2554{2558. [12] M. H usken and P. Stagge, Recurrent neural networks for time series classication, Neurocomputing, 50 (2003), pp. 223{235. [13] C. Jammazi, Finite-time partial stabilizability of chained systems, Comptes Rendus Mathematique, 346 (2008), pp. 975{980. [14] J.-S. Jang, S.-Y. Lee, and S.-Y. Shin, An optimization network for ma- trix inversion., in NIPS, 1987, pp. 397{401. [15] J. Kamruzzaman and S. M. Aziz, A note on activation function in multi- layer feedforward learning, in Neural Networks, 2002. IJCNN'02. Proceedings of the 2002 International Joint Conference on, vol. 1, IEEE, 2002, pp. 519{ 523. [16] S. Li, S. Chen, and B. Liu, Accelerating a recurrent neural network to nite-time convergence for solving time-varying sylvester equation by using a sign-bi-power activation function, Neural processing letters, (2013), pp. 1{17. Bibliography 42 [17] S. Li, Y. Li, B. Liu, and T. Murray, Model-free control of lorenz chaos using an approximate optimal control strategy, Communications in Nonlinear Science and Numerical Simulation, 17 (2012), pp. 4891{4900. [18] A. M. Lyapunov, The general problem of the stability of motion, Interna- tional Journal of Control, 55 (1992), pp. 531{534. [19] R. Manherz, B. Jordan, and S. Hakimi, Analog methods for computa- tion of the generalized inverse, IEEE Transactions on Automatic Control, 13 (1968), pp. 582{585. [20] W. S. McCulloch and W. Pitts, A logical calculus of the ideas imma- nent in nervous activity, The bulletin of mathematical biophysics, 5 (1943), pp. 115{133. [21] M. Minsky and S. Papert, Perceptrons: an introduction to computational geometry (expanded edition), 1988. [22] J. D. Roberts, Linear model reduction and solution of the algebraic riccati equation by use of the sign function, International Journal of Control, 32 (1980), pp. 677{687. [23] F. Rosenblatt, The perceptron: A probabilistic model for information stor- age and organization in the brain., Psychological review, 65 (1958), p. 386. [24] W. E. Roth, The equations ax-yb= c and ax-xb= c in matrices, Proceedings of the American Mathematical Society, 3 (1952), pp. 392{396. [25] H. Sak, A. Senior, and F. Beaufays, Long short-term memory recurrent neural network architectures for large scale acoustic modeling, in Fifteenth Annual Conference of the International Speech Communication Association, 2014. Bibliography 43 [26] M. Smith and M. Wisten, A continuous day-to-day trac assignment model and the existence of a continuous dynamic user equilibrium, Annals of Operations Research, 60 (1995), pp. 59{79. [27] P. Werbos, Beyond regression: New tools for prediction and analysis in the behavioral sciences, (1974). [28] B. Widrow et al., Adaptive" adaline" Neuron Using Chemical" memis- tors."., 1960. [29] L. Xiao and Y. Zhang, Zhang neural network versus gradient neural net- work for solving time-varying linear inequalities, IEEE transactions on neural networks, 22 (2011), pp. 1676{1684. [30] Y. Zhang and S. S. Ge, Design and analysis of a general recurrent neu- ral network model for time-varying matrix inversion, IEEE Transactions on Neural Networks, 16 (2005), pp. 1477{1490. [31] Y. Zhang, W. Ma, and B. Cai, From zhang neural network to newton iteration for matrix inversion, IEEE Transactions on Circuits and Systems I: Regular Papers, 56 (2009), pp. 1405{1415. [32] Y. Zhang and C. Yi, Zhang neural networks and neural-dynamic method, Nova Science Publishers, Inc., 2011.
Abstract (if available)
Abstract
The Sylvester equation is a kind of important equations in the control system, which has wide application in both engineering and science. Moreover, Recurrent Neural Network has been proven is an efficient way to solve the finite-time stable problem. In this thesis, we present a Recurrent Neural Network with tunable activation function to improve the performance of existing Recurrent Neural Network. We would show it advantages theoretically through the rate of convergence. Furthermore, we would show it has better robustness by two practical examples.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
The application of machine learning in stock market
PDF
An application of Markov chain model in board game revised
PDF
Supervised learning algorithms on factors impacting retweet
PDF
Elements of dynamic programming: theory and application
PDF
Linear differential difference equations
PDF
Neural matrix factorization model combing auxiliary information for movie recommender system
PDF
Interval arithmetic and an application in finance
PDF
Applications of contact geometry to first-order PDEs on manifolds
PDF
Tamed and truncated numerical methods for stochastic differential equations
PDF
Facial key points detection by convolutional neural network
PDF
Construction of orthogonal functions in Hilbert space
PDF
Max-3-Cut performance of graph neural networks on random graphs
PDF
Dynamic network model for systemic risk
PDF
Improvement of binomial trees model and Black-Scholes model in option pricing
PDF
Regularity problems for the Boussinesq equations
PDF
Dynamic approaches for some time inconsistent problems
PDF
Fractional calculus and its application
PDF
Validation of an alternative neural decision tree
PDF
Monochromatic exact solution to Maxwell's equations with finite energy in the vacuum
PDF
Stochastic differential equations driven by fractional Brownian motion and Poisson jumps
Asset Metadata
Creator
Zhang, Siqi
(author)
Core Title
Recurrent neural networks with tunable activation functions to solve Sylvester equation
School
College of Letters, Arts and Sciences
Degree
Master of Science
Degree Program
Applied Mathematics
Publication Date
07/17/2017
Defense Date
06/23/2017
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
dynamic system,OAI-PMH Harvest,recurrent neural network,Sylvester equation,tunable activation system
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Lotosky, Sergey (
committee chair
), Sacker, Robert (
committee member
), Tiruviluamala, Neelesh (
committee member
)
Creator Email
jimmy199389@gmail.com,siqiz@usc.edu
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c40-401108
Unique identifier
UC11266292
Identifier
etd-ZhangSiqi-5528.pdf (filename),usctheses-c40-401108 (legacy record id)
Legacy Identifier
etd-ZhangSiqi-5528.pdf
Dmrecord
401108
Document Type
Thesis
Rights
Zhang, Siqi
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
dynamic system
recurrent neural network
Sylvester equation
tunable activation system