University of Southern California Dissertations and Theses

Stochastic games with expected-value constraints

(USC Thesis Other)

Stochastic games with expected-value constraints

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Copyright 2020 Yunan Zhou Stochastic Games with Expected-Value Constraints by Yunan Zhou A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY December 2020 Acknowledgements I sincerely appreciate Professor Jong-Shi Pang, my academic advisor, for his patient guidance, powerful support and continuous encouragement from the bottom of my heart. Without his help, I couldn’t have accomplished this thesis. It is his kindness, wisdom, devotion and inspiration that keeps me improving all the time. His earnestness and persistence have a strong and far-reaching influence on me. I’m so lucky to be a student of Professor Pang and having a period to work with himisasignificanthonorformeinmyentirelife. Besides,I’dliketogivemysincerethankstoallthecommitteemembersattendingmydissertation defense,includingProfessorSuvrajeetSen,ProfessorRahulJain,ProfessorJohnGunnarCarlsson and Professor Meisam Razaviyayn. I really appreciate all the practical suggestions and helpful guidancetheygavetomeinimprovingmythesis. Furthermore, I’m more than grateful to all the people I met in the ISE department at USC. I want toshowmydeepestappreciationfortheirkindnessandhelp. Itisamemorableexperiencetowork togetherwiththem. Finally,IwouldliketothankmyparentsGangZhouandYanpingLufortheircontinuoussupport andencouragement. IbelievethisthesisisoneofthebestgiftsIcangivetotheminmywholelife. ii TABLEOFCONTENTS Acknowledgements ...................................... ii ListofTables.......................................... v ListofFigures ......................................... vi Abstract............................................. vii 1 Introduction ........................................ 1 2 LagrangianSchemeandItsConvergenceAnalysis ................... 9 2.1 ModelFormulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 AssumptionsandPreliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2.1 AssumptionsoftheLagrangianScheme . . . . . . . . . . . . . . . . . . . 11 2.2.2 ExistenceandUniquenessoftheN.E. . . . . . . . . . . . . . . . . . . . . 14 2.3 LagrangianSchemeDescription . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3.1 TheMax-MinGame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.3.2 VariationalInequalityProblemandItsEquivalences . . . . . . . . . . . . . 22 2.3.3 Outer-LoopOutline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.3.4 Inner-LoopOutline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.3.5 DescriptionoftheLagrangianScheme . . . . . . . . . . . . . . . . . . . . 37 2.4 Outer-LoopConvergenceAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.5 Inner-LoopConvergenceAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.5.1 Boundednessofthe Part . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.5.2 Inner-LoopConvergenceProof . . . . . . . . . . . . . . . . . . . . . . . . 49 3 ConvergenceRateAnalysisoftheLagrangianScheme................. 62 3.1 OutlinesoftheRateAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.2 Outer-LoopConvergenceRateAnalysis . . . . . . . . . . . . . . . . . . . . . . . 68 3.3 Inner-LoopConvergenceRateAnalysis . . . . . . . . . . . . . . . . . . . . . . . 73 3.3.1 Inner-LoopConvergenceRate . . . . . . . . . . . . . . . . . . . . . . . . 74 3.3.2 Inner-LoopImplementationDesign . . . . . . . . . . . . . . . . . . . . . 87 3.4 Non-ExpansivenessoftheProximalMapping . . . . . . . . . . . . . . . . . . . . 92 3.5 TheSchemeinPracticalImplementation . . . . . . . . . . . . . . . . . . . . . . . 95 iii 4 ⇢ MethodandItsConvergenceProperty......................... 100 4.1 FormulationandAssumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 4.1.1 Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 4.1.2 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.2 Descriptionofthe⇢ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.3 Inner-LoopConvergenceAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4.3.1 TheExistenceandUniquenessofInnerIteration’sSolution . . . . . . . . . 108 4.3.2 Inner-LoopConvergenceProof . . . . . . . . . . . . . . . . . . . . . . . . 110 4.4 Outer-LoopConvergenceAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5 NumericalResults ..................................... 122 5.1 GeneralModelDescriptionandReviewsoftheMethods . . . . . . . . . . . . . . 122 5.1.1 ModelDescription . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.1.2 ReviewoftheLagrangianScheme . . . . . . . . . . . . . . . . . . . . . . 125 5.1.3 Reviewofthe⇢ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5.2 Case1: 5⇥ 5⇥ 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 5.2.1 ModelIntroductionandGeneration . . . . . . . . . . . . . . . . . . . . . 132 5.2.2 Case1: LagrangianScheme . . . . . . . . . . . . . . . . . . . . . . . . . 135 5.2.3 Case1: ⇢ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 5.3 Case2: 8⇥ 8⇥ 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 5.3.1 ModelIntroductionandGeneration . . . . . . . . . . . . . . . . . . . . . 144 5.3.2 Case2: LagrangianScheme . . . . . . . . . . . . . . . . . . . . . . . . . 147 5.3.3 Case2: ⇢ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 5.4 Case3: 8⇥ 8⇥ 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 5.4.1 ModelIntroductionandGeneration . . . . . . . . . . . . . . . . . . . . . 153 5.4.2 Case3: ⇢ Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 5.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 6 FutureWork ........................................ 162 References ........................................... 164 iv List of Tables 1 N.E. solution in case 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 2 Dual solution in case 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 3 C’s tunning in case 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4 Increasing step-sizes under different variances in case 1 . . . . . . . . . . . . . . . 138 5 Simulation results of the Lagrangian scheme with c = 0.0001 in case 1 . . . . . . . 139 6 Simulation results of the⇢ method in case 1 . . . . . . . . . . . . . . . . . . . . . 143 7 N.E. solution in case 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 8 Dual solution in case 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 9 C’s tunning in case 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 10 Increasing step-sizes under different variances in case 2 . . . . . . . . . . . . . . . 149 11 Simulation results of the Lagrangian scheme with c = 0.001 in case 2 . . . . . . . . 150 12 Simulation results of the⇢ method in case 2 . . . . . . . . . . . . . . . . . . . . . 152 13 Generation methods of the model elements in case 3 . . . . . . . . . . . . . . . . . 155 14 One set of Slater’s points of all players in case 3 . . . . . . . . . . . . . . . . . . . 156 15 N.E. solution in case 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 16 Dual solution in case 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 17 Simulation results of the⇢ method in case 3 . . . . . . . . . . . . . . . . . . . . . 159 v List of Figures 1 The flowchart of the Lagrangian scheme rate analysis . . . . . . . . . . . . . . . . 67 vi Abstract Stochastic game is a combination of stochastic programming and game theory. This thesis studies the stochastic games with expected-value constraints where each player’s constraints are individual and some of them are described by inequalities involving expected-value func- tions. Inthedeterministicregimes,theargumentsin[13]provideaniterativealgorithmcalled the "Proximal Best-Response" (Proximal BR) scheme to obtain a Nash Equilibrium. Under some assumptions on the spectral property related to the Proximal BR mapping, the authors have shown the convergence property of the scheme. However, when we apply the Proximal BR scheme to solve the stochastic games, it requires us to have the exact evaluations of all expected-value functions, which is un-realistic. To solve the stochastic games with expected- valueconstraints,wedeveloptwomethods. (i)Thefirstoneiscalledthe"LagrangianScheme" that is a two-loops method. In this scheme, we assume that for each player, parameterized by others’ strategies, his/her sub-problem is a convex minimization problem with a twice differ- entiableobjectiveandthedifferentiableexpected-valueconstraints. Underthoseassumptions, firstly, for each player, we relax his/her expected-value constraints by augmenting the objec- tive with a weighted sum of the expected-value constraint functions and we call the weights used here as his/her dual variables. After these relaxations, for each player, his/her objective becomes a Lagrangian function that is convex in his/her decision variables and concave in his/herdualvariables. Now,givenothers’strategies,his/hersub-problembecomesamax-min problem. Hence, the relaxations make the stochastic game with expected-value constraints be amax-mingamewithdeterministicconstraints. Andsolvingthismax-mingameisequivalent to solving a Variational Inequality (V.I.) problem. With the monotonicity assumption on the equivalent V.I. problem, we solve it using an iterative algorithm called the "Proximal Point" (P.P.)methodwhosewholeiterationsformtheouter-loopofourLagrangianscheme. Whilein vii each iteration of the P.P. method, we need to solve a new V.I. problem, which is equivalent to solvingacorrespondingmax-mingameagain. Tosolvethismax-mingameineachiterationof theP.P.method,wedeveloptheinner-loopcombiningthe"Best-Response"(BR)algorithmand the "Sample Average Approximation" (SAA) method. In each inner iteration, we enlarge the size of independently and identically distributed (i.i.d.) random samples and use the SAA to relaxalltheexpected-valuefunctions. Then, byapplyingtheBR,weobtainthebest-response solutionforeachplayerinthecurrentinneriteration. Hence,theinner-loopsolutionssequence in each outer iteration is actually a sequence of the solutions of sampled max-min problems. (ii) The second method to solve the stochastic expected-value constrained games is called the "⇢ Method". It is also a two-loops algorithm where in each outer iteration, we fix the sample size which will increase to infinity as the outer-loop evolves. In each outer iteration, inspired by SAA, we first generate some new i.i.d. random samples as supplements and then use the sampled average functions to replace the expected-value functions in each player’s expected- value constraints. To avoid the infeasibility incurred by this replacement, for each player, we add a non-negative dummy variable ⇢ as the adjustment term to guarantee the feasibility of his/hersub-problem. Also,wepenalizethevalueof⇢ oneachplayer’sobjectivefunctionwith the penalizing coefficient being the current sample size. Thus, in each outer iteration of the ⇢ method, we solve a deterministic game with one more dummy variable in each player’s sub- problem. Under some assumptions on the spectral property related to the BR mapping, the inner-loop of the⇢ method is developed to solve the deterministic game formed in each outer iteration. In thisstudy, wehave shownthe convergenceproperties inalmosteverywhere (a.e.) sense for both the Lagrangian scheme and the ⇢ method. We also provide the convergence rate analysis of the Lagrangian scheme to enable us to control the error in expectation sense in the real implementation. Finally, we give some numerical results to illustrate some basic viii propertiesofthetwomethodsrespectively. ix 1 Introduction Historically,therewasalargeamountofresearchonstochasticprogramming(SP).Sincethestudy oflinearprogrammingwithuncertaintybyGeorgeB.Dantzigfromthemidof1950s[9],thetopic of stochastic programming has become one of the most interesting subjects in the optimization area. Many papers [20,22,23,28,29,39,40] have addressed and analyzed the theory and algo- rithmsofSPfromdifferentperspectives. Inaclassicalmodelofthetwo-stagestochasticprogram- ming problem, the first stage involves an optimization problem with an expected-value function in the objective. The value of this expected-value function is the expectation of the optimal value of a second-stage optimization problem whose constraints contain the first-stage decision vari- ables. Inothermodels,theobjectiveisadeterministicfunction,whiletheconstraintscontainsome expected-valuerandomfunctions[5]. One of the most difficulties in solving a stochastic programming problem is the exact evaluations ofexpected-valuefunctions,especiallywhentherearesomecontinuouslydistributedrandomvari- ables in the problem. Hence, to trigger the study and simplify the analysis of certain optimiza- tion problems having expected-value constraints, some assumptions are needed. For example, O’Brein [34] studied the case where the number of scenarios with positive probability is finite. Some other studies are built upon the assumptions on the properties of the expected-value func- tions. For example, in [26], the authors assumed that the random function is jointly convex in the decisionandrandomvariables. However,noneoftheabovestudiescandealwiththecasewherethereareseveralnon-cooperative agents. Whilewiththedevelopmentofindustryandthesocial-economicconnection,anincreasing number of such cases emerged. Therefore, to handle them, we need to take the non-cooperative 1 gametheoryintoourconsiderations. Non-cooperative game theory has a long history. Large parts of studies focus on the situations undercertainty. Inagamemodel,thereareN playerseachofwhomisselfishandmakesdecision aiming at maximizing his/her payoff that might be influenced by others’ strategies. Providing a solution concept of a game, a Nash Equilibrium (N.E.) [32,33] is a tuple of strategies where for eachplayer,he/shecannotbebetteroffbyunilaterallydeviatingfromhis/herequilibriumstrategy whentherivalskeeptheirownequilibriumstrategies. Bensoussan [2] was the first one noting that finding a N.E. can be formulated as solving a vari- ational inequality (V.I.) problem. Now this scheme was primarily developed, particularly for the computationoftheN.E.bysolvingthefinite-dimensionalV.I.andcomplementarity(CP)problems after the publication of the survey [12,19]. Two recent studies on obtaining Nash Equilibria via thevariationalapproachare[11,13]wheremanyreferencesareprovided,e.g.[1,3,24,25,30,38]. TherearealsosomeothermethodstoobtainaN.E.bysolvingaV.I.problem,e.g.[7,10,16,18]. Actually,by[13],wecanuseaniterativemethodtocomputetheN.E.withouttransferringthegame problemtoaV.I.problem. Indeed,sinceaN.E.canbecharacterizedasafixed-pointofaproximal- responsemapping,wecanusetheiterativealgorithmtoobtainaN.E.. Startingfromagivenpoint y 0 ,wegetthenextiteratefromthefixed-pointiteration:y k+1 , ˆ x(y k )fork=0,1,2,...,1. Inthis scheme,ithasbeenprovedthataslongastheproximalresponsemappingiscontractive,thesolu- tions sequence (y k ) 1 k=1 will converge to the unique N.E. of the corresponding game. In a weaker version, if the proximal response mapping is non-expansive and a N.E. exists, we can still deduce thesameconvergenceresultunderanaveragingschemesuchas: z k+1 =z k +⌧ k (ˆ x(z k ) z k )where thesequenceofpositivescalars {⌧ k }# 0satisfies 1P k=0 ⌧ k =1. In addition to the above methods, there are also some extensive studies of different algorithms to 2 computetheN.E.[31,42]. However, all the above schemes are not able to analyze the problems under uncertainty. Hence, some more powerful techniques are needed to deal with the games under uncertain scenarios. In this thesis, we will combine some SP algorithms and certain game theory techniques to develop somenewalgorithmsinsolvingthestochasticgameswithexpected-valueconstraints. Before we introduce the contributions of our work, let’s discuss some previous studies directly relatedtothecontentsinthethesis. The first study we need to mention is the work of Wei Wang. In her thesis [43], she studied the stochasticmin-maxprogrammingproblem. SheusedtheSAAalgorithmandtheboundingstrategy to compute the solution under a given confidence interval and provided the relevant convergence rateanalysis. Besides,inherpaper[44],shetalkedabouthowtousetheSAAalgorithmtogetthe solution of expected-value constrained programming problem accompanied with the convergence rate analysis. However, the work of Wang mainly focuses on the single agent stochastic program- mingproblem. Asforthedeterministicgameproblem,likewhathavebeenmentionedabove,wecanuseaproxi- malbest-responsealgorithmtogettheN.E.. Inthebook[13],theauthorssystematicallyintroduced theapplicationoftheproximalbest-responsealgorithmonfindingaN.E.ofadeterministicgame. In this scheme, it assumes for each player, the constraint is private and the individual objective function is twice continuously differentiable and the diagonal dominance (DD) assumption [13] holds. For each player, once we have regularized his/her objective function by adding the proxi- mal term, his/her sub-problem becomes strongly convex. The above modification combined with the mentioned assumptions enables us to apply the best-response algorithm with the guarantee of the sequential convergence to a N.E.. This algorithm just has one-loop. However, the study here 3 mainlyfocusesonthegameanalysisunderdeterministicsituation. Insuchasetting,someanalyses canbesimplified. Forexample,sincethereisnostochasticterminthegamemodel,e.g. expected- value function, we do not need to use the SAA method to approximate the expected-value func- tion in each iteration so that each objective remains the same in different iterations. Hence, once we have added the regularization terms, under some general assumptions, the solution sequence will possess the non-expansiveness property leading to the convergence result. However, if some expected-value functions exist in the objective, we need to use the SAA method to asymptotically approximatethembyenlargingthesamplesizeastheiterationgoeson. Thiskindofhandlingwill change the objective function when the algorithm iterates. Thus, the use of fixed-point theorem to show the convergence property will be invalid. From this point, we can see that once we have incorporated the stochastic terms into our model, making it stochastic, some methods used in the analysisofthedeterministicgameproblemwillbeinadequate. Unlike the topic of stochastic programming and deterministic game theory, both of which have been studied over 50 years, stochastic game theory is still at its infant stage. Today, with those two topics being much mature, the opportunity to study the combination of them has been ripe. One recent study preceding this thesis is [35]. In this paper, each player’s problem is actually a two-stage SP problem [5] parameterized by other players’ strategies, where the first stage is an optimization problem with the objective function having an expected-value term called the “re- coursefunction”whichequalstheoptimalvalueofthesecond-stageoptimizationproblem. Inthis paper, the authors developed a one-loop algorithm. In each iteration, the algorithm combines the methodsofproximalbest-responseandSAA.Underseveralassumptions,e.g. DDassumptionand theuniformlawoflargenumbers,theauthorsshowedthatanyaccumulationpointofthesequence generatedbysolvingtheproximalbest-responseproblemsinalliterations,mustbetheN.E.ofthe 4 originalgameinalmosteverywhere(a.e.) sense. Another paper studying stochastic games which is highly relevant to this thesis is [27]. Like the paper of [35], in [27], the authors mainly focused on the stochastic games where an expected- value function is included in each player’s objective. However, the main difference is that in [27], the expected-value function in each player’s objective is no longer a recourse function, instead, it is a one-stage function. In the algorithms, being different from the one-loop algorithm pro- posed in [35], some two-loops methods were developed in [27]. The main differences among those methods mentioned in [27] are in the updating scheme in each outer iteration while the es- sential frameworks are the same. In the basic algorithmic structure, the outer-loop is conceptually formedbytheproximalbest-responseiterationsundercertainty,whileeachouteriterationsolution is inexact. The inner-loop aims at using the generated i.i.d. random samples to approximate the expected-value function embedded in each player’s objective to obtain an inexact solution. One noteworthypointhereisthatintheinner-loop,theauthorsusedtheonlinegradientdecentmethod to generate the solutions sequence which can be proved to converge to a point being ✏-close to the real solution in the expectation sense. Compared with the SAA method, the advantage of ap- plying this scheme lies in the point that each time when we generate a new sample, we will have one update. Unlike SAA where we cannot do the computations until all the needed samples have been generated, the online gradient decent method enables us to lower the bar of triggering the computation. In [27], for each concrete algorithm, after describing the details of both outer and inner-loops, the authors proved the convergence property of the outer-loop and then gave the rate andtheoveralliterationcomplexityanalysesbywhichtheauthorseventuallyprovidedthespecific schemetoachievea✏-closesolutionintheexpectationsense. In this thesis, we will focus on the game where each player’s constraints have some expected- 5 value functions. In this specific topic, the authors of [36] have made certain interesting analyses. In [36], the authors studied several types of stochastic games, such as the stochastic games with each player’s objective including the expected-value function and constraints being deterministic and parameterized by others’ strategies. In the last part of this paper, the authors considered the stochastic games with expected-value constraints similar to the game problems we will inves- tigate in this thesis. However, throughout the whole paper, the authors only focused on proving the existence of the N.E. and describing the N.E.’s characterizations without pointing out how to compute them. With the significant developments of both stochastic programming and deterministic game theory and the inspirations given by the previous studies on the stochastic games, in this thesis, we will focus on developing the implementable computing algorithms in solving the stochastic games with expected-value constraints. To my best knowledge, the work in this thesis is the first one studying on this specific topic. Contributions: In this thesis, we study the stochastic games with expected-value constraints and develop two al- gorithms. The descriptions with more details are stated separately below. (1) Lagrangian Scheme: In this scheme, we use the Lagrangian multipliers to relax the expected- value constraints onto the objective for each player, which converts the game into a max-min game with deterministic constraints. Then, we develop a two-loops algorithm to solve this max-min game. In each iteration of the outer-loop, we actually solve a regularized max-min game which is equivalent to a V.I. problem due to the private properties of each player’s constraints. Using the convergence property of the proximal point method introduced in [12], we can easily establish the 6 convergenceresultfortheouter-loopsolutionssequence. Undereachouteriteration,wedeveloptheinner-looptosolvetheregularizedmax-mingame. We combine the best-response algorithm and the SAA method to obtain the inner-loop solutions se- quence which can be proved to converge to the unique N.E. of the regularized max-min game in almosteverywhere(a.e.) sense. In addition to the convergence results, we give the convergence rate analysis to figure out, given a positive value✏> 0, how to design the practical implementation to ensure that we can obtain a solutionthatis✏-closetooneN.E.oftheoriginalgameintheexpectationsense. (2) We also develop another scheme called the "⇢ Method". In this method, for each player, in- steadofrelaxingtheexpected-valueconstraintsandputtingthemwiththeirLagrangianmultipliers togetherintotheobjectivefunction,weuseSAAdirectlytoapproximatetheexpected-valuefunc- tionsintheconstraints. Toavoidthepotentialinfeasibilityineachplayer’ssub-problemafterthese approximations, for each player, we add a non-negative variable ⇢ to the right hand side (RHS) in each inequality whose left hand side (LHS) involves a sample averaged function. Meanwhile, to boundthevalueof⇢ ,wepenalizeitintheplayer’sobjectivefunction. Toguaranteetheasymptotic feasibility, the penalty coefficient equals the sample size used in SAA. Thus, in the inner-loop of the ⇢ method, we solve a deterministic SAA reformulated game. While, in the outer-loop, we get asequenceofN.E.solutionswhichisexpectedtoapproachoneN.E.oftheoriginalgameina.e.. It turns out that under the settings above, we can prove the convergence properties for both outer andinner-loopsofthe⇢ method. (3)Weconductthesimulationsinboththetwomethodstoshowtheirperformancesinsolvingthe stochasticgameswithexpected-valueconstraints. Wealsocomparethemandgiveourrecommen- dations of the selections of them in solving the stochastic games with expected-value constraints 7 underdifferentsituations. ThesisOutline: Thethesisisorganizedasbelow. In chapter 2, we give the formulation and assumptions in the Lagrangian scheme. Also, we prove itsconvergencepropertiesinbothofitsouterandinner-loops. Inchapter3,weprovidetheconvergencerateanalysisoftheLagrangianscheme. Thispartismore technical than the convergence analysis. To achieve the complete rate analysis, we accomplish both the outer and inner-loops’ rate studies and combine them together. Eventually, we show the schemeunderwhichwecandevelopourpracticalimplementationtohavea✏-closesolutioninthe expectationsense. In chapter 4, we introduce the second method named the "⇢ Method" and show its convergence propertiesunderthespecificsetofassumptions. In chapter 5, we report some numerical results to check the performances of both algorithms and thencomparethemtohavetheintuitionsoftheirprosandconsrespectively. In chapter 6, we discuss some interesting future works that can extend the current theoretical results. 8 2 LagrangianSchemeandItsConvergenceAnalysis In this chapter, we propose a two-loops algorithm called the “Lagrangian Scheme” to solve the stochastic games with expected-value constraints and then provide the proofs of its convergence propertiesinouterandinner-loopsrespectively. Thecontentsinthischapterareorganizedasfollows. In section 2.1, we formulate the stochastic game model and formally give the mathematical def- inition of the Nash Equilibrium (N.E.). In section 2.2, we provide the assumptions used in the wholechapterandtalkaboutthebasicpropertiesofthemodelweareconsideringhere. Insection 2.3,wetransfertheoriginalgametoanequivalent max-mingameandintroducethedetailsofthe Lagrangian scheme. In section 2.4, we prove the convergence property of the outer-loop. And lastly,insection2.5,weprovidetheproofoftheinner-loopconvergence. 2.1 ModelFormulation Inthissection,weformulatethemodelconsideredintheLagrangianscheme. Consider a non-cooperative game with N players, labeled as i =1, ..., N. For each player i, his/her strategy is represented by a n i -dimensional vector x i 2R n i and there is an objective function ✓ i (x i ,x i ) whose value depends on all players’ strategies x, (x i ) N i=1 . We assume that ✓ i (x i ,x i ) is twice differentiable. A private strategy set X i ✓ R n i that is compact and convex is given. Define X , N Q i=1 X i and X i , Q j6=i X j . Thus, we have X ✓ R n with n = N P i=1 n i and X i ✓ R n n i . To include the stochastic elements in each player’s sub-problem, we introduce a R d -dimensional random vector ! defined on the probability space (⌦ ,F,IP) where⌦ is the sample space, F is 9 the -field generated by the subsets of⌦ and IP is a probability measure defined on F [14]. We assumethatallplayershavetheaccesstoobtainthecompleteinformationof!. Thus,therandom vector ! is shared by all of them. However, since ! is a random vector, such assumption can still cover the case where the sets of random variables used in different players’ sub-problems are disjointtoeachother. In each playeri’s objective, we have a random function H i : R n ⇥ R d 7! R which, givenx i 2 X i ,isstronglyconvexinx i 2X i withafixedmodulusunderanyrealized!. Weassumethatthe expectationofH i (x i ,x i ,!),e.g. E[H i (x i ,x i ,!)],existsforanyx2Xandequals✓ i (x i ,x i ). In playeri’sconstraints,wehaveavector-valuedrandomfunctionG i :R n i ⇥ R d 7! R k i whereeach component function of G i (x i ,!), e.g. G i j (x i ,!) withj=1,2,3,...,k i , is convex inx i under any realized !. We assume that the expectation ofG i (x i ,!), e.g. E[G i (x i ,!)], exists for anyx i 2X i andthusplayeri’sfeasibleregionisrepresentedas {x i 2X i |E[G i (x i ,!)] 0}. Now,thegamecanbeformulatedasbelow: 8 > > < > > : min x i 2 X i ✓ i (x i ,x i ),E[H i (x i ,x i ,!)] s.t. E[G i (x i ,!)] 0 9 > > = > > ; N i=1 (2.1) withG i (x i ,!)beingak i -dimensionalvectorforanyi,e.g. G i (x i ,!)= 2 6 6 6 6 6 6 6 6 6 6 4 G i 1 (x i ,!) G i 2 (x i ,!) . . . G i k i (x i ,!) 3 7 7 7 7 7 7 7 7 7 7 5 . A Nash Equilibrium (N.E.) in game (2.1) is a tuple x ⇤ , (x ⇤ ,i ) N i=1 2X , N Q i=1 X i such that 10 x ⇤ ,i 2arg min x i 2 Z i ✓ i (x i ,x ⇤ , i )whereZ i , {x i 2X i |E[G i (x i ,!)] 0}foranyi=1,···,N. Intheabovemodel,wecanseethattheconstraintsineachplayer’ssub-problemareprivateinthe sense that for each player i, the other players’ strategies, e.g. x i , are not parameterized into i’s constraintsinanyform. 2.2 AssumptionsandPreliminaries In the following sections of this chapter, we will develop an algorithm named the “Lagrangian Scheme”tosolvethegamedescribedintheabovesection. Beforeweintroducethisschemeindetail,let’slistalltheassumptionsaboutthemodelusedinthe latteranalyses. 2.2.1 AssumptionsoftheLagrangianScheme TheassumptionsintheLagrangianschemearedividedinto3groupsas(A1),(A2)and(A3). The specificdetailsofthoseassumptionsareshownbelow. AssumptionsintheLagrangianScheme: (A1): Intheoriginalgame(2.1),foreachplayeri,wehavetheassumptionsasfollows. (A1.1): TheprivatesetX i isconvexandcompact. (A1.2): Givenx i 2X i ,H i (x i ,x i ,!)isstronglyconvexinx i 2X i withafixed modulusunderanyrealized! 1 . Also,H i (x i ,x i ,!)istwicedifferentiable inbothx i andx i overanopensetcontainingthedomainX = N Q i=1 X i forany realized!. 1 A function f : X 7! R is strongly convex in x 2X with modulus µ if for any x 2X and y 2X, we have f(↵x +(1 ↵y )) ↵f (x)+(1 ↵ )f(y) ↵ (1 ↵ )µ 2 kx yk 2 forall↵ 2[0,1]. 11 (A1.3): Underanyrealized!,foranycomponentfunctionofG i (x i ,!), e.g. G i j (x i ,!)forj2{ 1,2,...,k i }, G i j (x i ,!)isconvexinx i andcontinuously differentiableinx i overanopensetcontainingX i . (A1.4): Intheobjective,alltheE[H i (x i ,x i ,!)],E[r x H i (x i ,x i ,!)]andE[r 2 x H i (x i ,x i ,!)] existand✓ i (x i ,x i ),E[H i (x i ,x i ,!)]istwicedifferentiableinxwiththeequalities r x ✓ i (x i ,x i ),r x E[H i (x i ,x i ,!)] =E[r x H i (x i ,x i ,!)]andr 2 x ✓ i (x i ,x i ), r 2 x E[H i (x i ,x i ,!)] =E[r 2 x H i (x i ,x i ,!)]holdingforanyxoveranopen setcontainingX. Also,intheconstraints,bothE[G i (x i ,!)]andE[r x iG i (x i ,!)] existandE[G i (x i ,!)]iscontinuouslydifferentiableinx i withtheequality r x iE[G i (x i ,!)] =E[r x iG i (x i ,!)]holdingforanyx i overanopenset containingX i . (A1.5): BoththerandomfunctionsH i (x i ,x i ,!)andr x H i (x i ,x i ,!)satisfythe uniformlawoflargenumbersoverX 2 . Also,boththeG i (x i ,!)andthematrix-valued randomfunctionr x iG i (x i ,!)satisfytheuniformlawoflargenumbersoverX i . (A1.6): Slater’sconditionholdsfortheconstraints: {x i 2X i | E[G i (x i ,!)] 0}, e.g.9 ˆ x i 2ri(X i ) 3 suchthatE[G i (ˆ x i ,!)]< 0. (A2): DiagonalDominance(DD)Condition[13]: Foreachplayeri,wedefine: (1) ii , inf x2 X smallesteigenvalueofr 2 x i ✓ i (x)2(0,1), 2 A random functionT : X⇥ R d 7! R n orT : X⇥ R d 7! M n1⇥ n2 whereM n1⇥ n2 represents the space ofn 1 ⇥ n 2 matrices satisfies the uniform law of large numbers overX ifP{ lim L!1 [sup x2 X || 1 L L P s=1 T(x,! s ) E[T(x,!)]|| 2 ]=0} = 1 where w , (! s ) 1 s=1 is a realized random vector sequence with ! s being independently generated and having the identical distribution for each s. For the matrix-valued element, the norm operator || · || 2 here represents the correspondinginducedl 2 -norm. 3 Wesaythatapointxisarelativeinteriorpointofanon-emptyconvexsetX,e.g. x2ri(X),ifx2Xandthereexists anopenballScenteredatxsuchthatS\ aff(X)⇢ Xwhereaff(X)representstheaffinehullofsetX[4]. 12 (2) ij , sup x2 X r 2 x i x j ✓ i (x) 2 <1,forallj6=i; Thenthereexistsasetofpositiveconstants (d i ) N i=1 suchthat ii d i > P j6=i ij d j 8i=1,2,3,...,N. (A3): ThefunctionF(Z)= 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 r x 1(✓ 1 (x 1 ,x 1 )+( 1 ) T E[G 1 (x 1 ,!)]) r 1(✓ 1 (x 1 ,x 1 )+( 1 ) T E[G 1 (x 1 ,!)]) . . . r x N(✓ N (x N ,x N )+( N ) T E[G N (x N ,!)]) r N(✓ N (x N ,x N )+( N ) T E[G N (x N ,!)]) 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 withZ = 2 6 6 6 6 6 6 6 6 6 6 4 x 1 1 x 2 2 . . . x N N 3 7 7 7 7 7 7 7 7 7 7 5 ismonotone 4 intheregionwherex i 2X i and i 0foralli’s. Remark 1: It is routine to check that under the assumption (A1), the assumption (A3) is implied bythefollowingassumption: ( ¯ A3): Thefunction ¯ F(X)= 2 6 6 6 6 6 6 4 r x 1✓ 1 (x 1 ,x 1 ) . . . r x N✓ N (x N ,x N ) 3 7 7 7 7 7 7 5 withX = 2 6 6 6 6 6 6 6 6 6 6 4 x 1 x 2 . . . x N 3 7 7 7 7 7 7 7 7 7 7 5 ismonotoneintheregionwherex i 2X i foralli’s. For each i, both assumptions (A1) and (A2) actually imply the strong convexity of ✓ i (x i ,x i ) in x i 2X i given any x i 2X i . And also, for each player i, the assumption (A1) implies the 4 A vector-valued function F : K✓ R n 7! R n is said to be monotone, if (X Y) T (F(X) F(Y)) 0 for any X2K andY 2K. 13 convexityofeachofhis/herexpected-valueconstraints’componentfunction,e.g. E[G i j (x i ,!)]for j2{ 1,2,...,k i },inx i 2X i . What’s more, the assumptions (A1) and (A2) together imply the existence and uniqueness of the N.E.ingame(2.1). Even though the conclusions of existence and uniqueness of the N.E. in game (2.1) are implied in [13] provided that both (A1) and (A2) hold, it is still not trivial to check them directly. We will givetheproofsinthenextsub-sectionformally. 2.2.2 ExistenceandUniquenessoftheN.E. Inthissub-section,wewillshowthatintheoriginalgame(2.1),theN.E.existsandisuniqueifthe assumptions(A1)and(A2)hold. Beforeweshowtheproof,let’sgiveonedefinitionfirst. Definition1: Given a positive vector d=(d 1 ,d 2 ,...,d n ) T 2R n + , |·| d is a norm on the R n -space with |x| d , max i=1,2,...,n |x i | d i foranyn-dimensionalvectorx=(x 1 ,x 2 ,...,x n ) T inR n . Remark2: Fromthedefinition1,wecanseethat|·| d isamonotonicnorm 5 foranypositivevector d2R n . Now,let’sgivetheproofinthefollowinglemma. Lemma1 In game (2.1), suppose assumptions (A1) and (A2) hold. Then the N.E. of the game (2.1)existsandisunique. 5 Anormk·kdefinedontheR n -spaceismonotonicifkxkk ykforallx2R n andy2R n with |x|| y|. 14 Proof: Foreachplayeri,duetotheassumption(A1.6),theplayeri’ssub-problemingame(2.1)isfeasible. Also, based on (A1), the non-empty feasible setZ i , {x i 2X i |E[G i (x i ,!)] 0} is convex and compact. Starting fromµ=0 and selecting a pointx 0 , (x 0,i ) N i=1 such thatx 0,i 2Z i for alli’s, we define thepoint (x µ+1,i ) N i=1 foreachµiterativelyasbelow. Foreachi,wehave: x µ+1,i , arg min x i 2 Z i ✓ i (x i ,x µ, i ). Since✓ i (x i ,x µ, i )isstronglyconvexinx i 2Z i forallµ’sandi’s,wecanguaranteethattheabove solutioniswell-defined. For each i, under any µ 1, using the first order condition in convex optimization problem, we have: (x i x µ+1,i ) T r x i✓ i (x µ+1,i ,x µ, i ) 0 8x i 2Z i . (2.2) Henceforalli’s,wehave: (x µ,i x µ+1,i ) T r x i✓ i (x µ+1,i ,x µ, i ) 0. Applyingthesamelogicsinµ 1,wecanobtain: (x µ+1,i x µ,i ) T r x i✓ i (x µ,i ,x µ 1, i ) 0. Foreachi,combiningtheabovetwoinequalitiestogether,wehave: (x µ+1,i x µ,i ) T (r x i✓ i (x µ+1,i ,x µ, i )r x i✓ i (x µ,i ,x µ 1, i )) 0. ApplyingthemeanvaluetheoremontheLHSintheaboveinequality,wecanobtainthefollowing inequalitiesforeachi: 15 ii kx µ+1,i x µ,i k 2  ( P j6=i ij kx µ,j x µ 1,j k)kx µ+1,i x µ,i k )kx µ+1,i x µ,i k P j6=i ij ii kx µ,j x µ 1,j k where ij ’saredefinedinassumption(A2)foralli’sandj’s. Concatenatingtheinequalitiesforalli’s,wehave: 0 B B B B B B @ kx µ+1,1 x µ,1 k . . . x µ+1,N x µ,N 1 C C C C C C A  ˆ ⌥ 0 B B B B B B @ kx µ,1 x µ 1,1 k . . . x µ,N x µ 1,N 1 C C C C C C A (2.3) where ˆ ⌥= 0 B B B B B B B B B B @ 0 12 11 ··· 1N 11 21 22 0 ··· 2N 22 . . . . . . . . . . . . N1 NN ··· ··· 0 1 C C C C C C C C C C A . Let d , (d i ) N i=1 denote the set of positive constants being defined in assumption (A2). As both sidesintheinequality(2.3)arenon-negative,usingthemonotonicityofthenorm|·| d ,wehave: 0 B B B B B B @ kx µ+1,1 x µ,1 k . . . x µ+1,N x µ,N 1 C C C C C C A d  ˆ ⌥ d · 0 B B B B B B @ kx µ,1 x µ 1,1 k . . . x µ,N x µ 1,N 1 C C C C C C A d . 16 Assumption(A2)implies , ˆ ⌥ d < 1. Astheresult,wehave: 0 B B B B B B @ kx µ+1,1 x µ,1 k . . . x µ+1,N x µ,N 1 C C C C C C A d  · 0 B B B B B B @ kx µ,1 x µ 1,1 k . . . x µ,N x µ 1,N 1 C C C C C C A d . DefineD 1 , 0 B B B B B B @ kx 1,1 x 0,1 k . . . x 1,N x 0,N 1 C C C C C C A d . Thenforanyµ 1,wehave: 0 B B B B B B @ kx µ+1,1 x µ,1 k . . . x µ+1,N x µ,N 1 C C C C C C A d  µ D 1 . (2.4) Defined m , max i=1,2,...,N d i . From(2.4),foreachµ 1,wehavethefollowinginequalities: (2.4) =) max i=1,2,...,N kx µ+1,i x µ,i k d i  µ D 1 =) kx µ+1,i x µ,i k d m  µ D 1 8i=1,2,...,N =)kx µ+1,i x µ,i k µ D 1 d m 8i=1,2,...,N. (2.5) Duetothefactthat 0<< 1,from(2.5),foreachplayeri,itcanbeeasilyverifiedthat (x µ,i ) 1 µ=1 is a Cauchy sequence in theR n i -space with thel 2 -norm being the metric 6 . Due to the complete- 6 A sequence (x µ ) 1 µ=0 ✓ R n is a Cauchy sequence inR n with thel 2 -norm being the metric if for any positive value✏, thereexistsapositiveintegerN ✏ suchthatkx µ1 x µ2 k<✏aslongasµ 1 >N ✏ andµ 2 >N ✏ . 17 ness of theR n i -space 7 , we can conclude that (x µ,i ) 1 µ=0 must converge. Also, since Z i is a closed subset in R n i , the limit point of (x µ,i ) 1 µ=0 , e.g. x 1 ,i , will be in Z i . Hence the whole sequence ((x µ,i ) N i=1 ) 1 µ=0 willconvergeto (x 1 ,i ) N i=1 2Z, N Q i=1 Z i . Foreachi,passingthelimitintheLHSoftheinequality(2.2)andusingthecontinuityofr x i✓ i (x i ,x i ) inx2Xthatisimpliedbyassumption(A1.4)or(A2),wehave: (x i x 1 ,i ) T r x i✓ i (x 1 ,i ,x 1 , i ) 0 8x i 2Z i . Therefore, (x 1 ,i ) N i=1 is one N.E. of the game (2.1), which means the existence of the N.E. in (2.1) canbeguaranteed. SupposewehavetwodifferentN.E.’softhegame(2.1),e.g. (x 1 ,i ) N i=1 and(¯ x 1 ,i ) N i=1 . Theninboth thetwopoints,foreachi,usingthefirstorderconditions,wehave: (¯ x 1 ,i x 1 ,i ) T r x i✓ i (x 1 ,i ,x 1 , i ) 0 and (x 1 ,i ¯ x 1 ,i ) T r x i✓ i (¯ x 1 ,i ,¯ x 1 , i ) 0. Combiningthemtogetherandusingthesimilaranalysisintheexistenceproof,wehave: 0 B B B B B B @ kx 1 ,1 ¯ x 1 ,1 k . . . x 1 ,N ¯ x 1 ,N 1 C C C C C C A d  0 B B B B B B @ kx 1 ,1 ¯ x 1 ,1 k . . . x 1 ,N ¯ x 1 ,N 1 C C C C C C A d with 0<< 1, which contradicts to the assumption that (x 1 ,i ) N i=1 6=(¯ x 1 ,i ) N i=1 . Hence the N.E. ofthegame(2.1)isunique. ⇤ 7 AmetricspaceXiscompleteifeveryCauchysequenceinXconvergesanditslimitisinX. 18 Remark3: Intheaboveproof,wecanseethattheassumption(A2)playsasignificantrole,while theassumption(A1)isonlyusedtomakesurethatthefeasibleregionZ i isnon-empty,closedand convexforeachi,whichisimpliedby(A1.1),(A1.2)and(A1.6)together. With those assumptions from (A1) to (A3) and the conclusion of lemma 1, we can now describe theLagrangianschemetosolvethegame(2.1). 2.3 LagrangianSchemeDescription To solve the original game (2.1), a method named the "Lagrangian Scheme" is developed. As implied by its name, in the Lagrangian scheme, we resort to the Lagrangian function to solve game (2.1). Specifically, we need to use the Lagrangian function of each player’s sub-problem andcorrespondinglytransfereachplayer’ssub-problemtoa max-minproblem. Thus,theoriginal game (2.1) will be transferred to a max-min game. In the following sub-sections, we will show howtomakesuchtransformationandhowtosolvethegeneratednewgame. 2.3.1 TheMax-MinGame To solve the game (2.1), for each player i, we try to relax his/her expected-value constraints by considering the Lagrangian function and then convert the original game to a max-min game. The convertedmax-mingameisshownbelow: ⇢ max i 0 min x i 2 X i L i (x i ,x i , i ), ✓ i (x i ,x i )+( i ) T E[G i (x i ,!)] N i=1 . (2.6) 19 The N.E. of the above max-min game is a tuple (x ⇤ , ⇤ ) , (x ⇤ ,i , ⇤ ,i ) N i=1 such that x ⇤ ,i 2X i , ⇤ ,i 0 and ⇤ ,i 2argmax i 0 min x i 2 X i L i (x i ,x ⇤ , i , i ) with x ⇤ ,i 2arg min x i 2 X i L i (x i ,x ⇤ , i , ⇤ ,i ) for any i=1,···,N. For each player i, the function L i (x i ,x i , i ) is called the “Lagrangian Function” of i. Given all others’ strategies x i 2X i , L i (x i ,x i , i ) is linear in i and strongly convex in x i when i is non-negative. Defining⇤ i ,R k i + ,⇤ i , Q j6=i ⇤ j and⇤ , N Q i=1 ⇤ i ,wehave ( i ) N i=1 2⇤ . Givenany(x 0 i ) N i=1 2Xand i 0,foreachi,theoptimalsolutionoftheproblem min x i 2 X i L i (x i ,x 0 i , i ) is attainable and unique due to the strong convexity of ✓ i (x i ,x 0 i ) in x i 2X i and the convexity ofE[G i (x i ,!)] inx i 2X i . Also, letg i ( i ;x 0 i ), min x i 2 X i L i (x i ,x 0 i , i ) denote the dual function parameterizedbyx 0 i . Thenfromassumption(A1.6),wecanseethattheoptimalsolutionsofthe problem max i 0 g i ( i ;x 0 i ) are attainable as well. Hence, the use of max-min, instead of sup-inf, is meaningfulhere. Here, we hope we can find the relation between the N.E. of the game (2.1) and the N.E. of the game(2.6). Formally,forthispoint,wedevelopthefollowinglemma. Lemma2 In the game (2.1), suppose assumptions (A1) and (A2) hold. Then there exists at least one( ⇤ ,i ) N i=1 2⇤ suchthat(x ⇤ ,i , ⇤ ,i ) N i=1 isaN.E.ofthegame(2.6)with(x ⇤ ,i ) N i=1 beingtheunique N.E. of the game (2.1). Reversely, thex part of every N.E. in the game (2.6) is the unique N.E. of thegame(2.1). Proof: Wewillprove“)”first. Let (x ⇤ ,i ) N i=1 denotetheuniqueN.E.ofthegame(2.1). Foreachi,wehavethatx ⇤ ,i istheuniqueoptimizerofthei’ssub-problem: min x i 2 Z i ✓ i (x i ,x ⇤ , i ). From the assumption (A1.6), we know that the Slater’s condition holds in this sub-problem. Hence, by 20 referring to [6], we can deduce that there exists ⇤ ,i such that (x ⇤ ,i , ⇤ ,i ) satisfies the KKT condi- tions[6]. Therefore,wehave: k i P j=1 ⇤ ,i j E[G i j (x ⇤ ,i ,!)] = 0 (2.7) and x ⇤ ,i =argmin x i 2 X i L i (x i ,x ⇤ , i , ⇤ ,i ). (2.8) The dual function g i ( i ;x ⇤ , i ) parameterized by x ⇤ , i has been defined above. From the defini- tion ofg i ( i ;x ⇤ , i ) and the equality (2.8), we haveg i ( ⇤ ,i ;x ⇤ , i )= L i (x ⇤ ,i ,x ⇤ , i , ⇤ ,i ). Using the equality(2.7),wegetg i ( ⇤ ,i ;x ⇤ , i )= ✓ i (x ⇤ ,i ,x ⇤ , i )implyingthat ⇤ ,i 2argmax i 0 g i ( i ;x ⇤ , i )due totheweekduality[6]. Hence,wehaveforeachi,thereexists ⇤ ,i 0suchthat: ⇤ ,i 2argmax i 2 0 min x i 2 X i L i (x i ,x ⇤ , i , i )withx ⇤ ,i 2arg min x i 2 X i L i (x i ,x ⇤ , i , ⇤ ,i ). Therefore, (x ⇤ ,i , ⇤ ,i ) N i=1 isaN.E.ofthegame(2.6). Andthus,the“)”hasbeenproved. Reversely, suppose (x ⇤ ,i , ⇤ ,i ) N i=1 is a N.E. of the game (2.6). Then for each i, ⇤ ,i is the optimal dual solution of the i’s sub-problem: min x i 2 Z i ✓ i (x i ,x ⇤ , i ). Again, by referring to [6], we know that (x 0 ,i , ⇤ ,i ) satisfies the KKT conditions where x 0 ,i is the unique optimal solution of the i’s sub- problem. FromtheKKTconditions,wecandeducethatx 0 ,i =argmin x i 2 X i L i (x i ,x ⇤ , i , ⇤ ,i ). However,since(x ⇤ ,i , ⇤ ,i ) N i=1 isaN.E.ofthegame(2.6),wehavex ⇤ ,i =argmin x i 2 X i L i (x i ,x ⇤ , i , ⇤ ,i ) aswell. Using the strong convexity of L i (x i ,x ⇤ , i , ⇤ ,i ) inx i 2X i , we havex ⇤ ,i = x 0 ,i . Hence, for eachi, x ⇤ ,i istheuniqueoptimalsolutionofthei’ssub-problem,whichmeans (x ⇤ ,i ) N i=1 istheuniqueN.E. ofthegame(2.1). ⇤ 21 The above lemma actually gives us the equivalence between the game (2.1) and the game (2.6). Hence,tosolvethegame(2.1),itsufficestofocusonsolvingtheconvertedmax-mingame(2.6). However,solvingthegame(2.6)isnottrivial. Todoit,wewillintroducethe“VariationalInequal- ity”(V.I.)probleminthenextsub-sectionwhereweshowthatthe max-mingame(2.6)isactually equivalenttoaV.I.problem. Andthen,wecanusetheV.I.techniquestosolvethegame(2.6). 2.3.2 VariationalInequalityProblemandItsEquivalences ByreferringtoP2in[12],thedefinitionoftheV.I.problemisgivenbelow. Definition2: Given a subset K ofR n and a mapping F : K 7! R n , the variational inequality (V.I.) problem, denotedasVI(K,F),istofindavectorx2K suchthat (y x) T F(x) 0, 8y2K. ThesetofsolutionsinthisproblemisdenotedasSOL(K,F). From the above definition, we can see that whenK✓ R n is a closed convex set andF =r x f(x) withf(x)beingadifferentiableconvexfunctioninx2R n ,theproblemVI(K,F)isequivalentto finding a point satisfying the optimality first order condition of the convex optimization problem: min x2 K f(x)withSOL(K,F)denotingthesetofitsoptimalsolutions. However, to associate a max-min problem with a V.I. problem, we need to define a saddle point problemasthebridge. ByreferringtoP21in[12],wehavethefollowingdefinition. Definition3: 22 Let L: R n+m 7! R denote a real-valued function. Let X ✓ R n and Y ✓ R m be two given closed sets. The saddle problem associated with this triple (L,X,Y) is to find a pair of vectors (x,y)2X⇥ Y,calleda“saddlepoint”,suchthat: L(x,v) L(x,y) L(u,y) 8(u,v)2X⇥ Y. From [12], we know when bothX andY are convex andL(x,y) is continuously differentiable in (x,y) over an open set containingX⇥ Y and “convex-concave” inx2X andy2Y, the saddle problem will be equivalent to a V.I. problem. Here, we sayL(x,y) is “convex-concave” ifL(·,y) is convex for each fixed but arbitrary y 2Y and L(x,·) is concave for each fixed but arbitrary x2X. Formally,wehavethat: if L(x,y) is continuously differentiable in (x,y) over an open set containing X ⇥ Y and also “convex-concave” in x and y and both X and Y are closed convex sets, then (x,y) is a saddle pointoftheproblem (L,X,Y)ifandonlyif(iff) (x,y)solvestheVI(X⇥ Y,F)where F(u,v), 0 B B @ r u L(u,v) r v L(u,v) 1 C C A , (u,v)2X⇥ Y. The above conclusion can be easily verified by invoking the respective minimum and maximum principlesforthetwoproblems: min u2 X L(u,y)and max v2 Y L(x,v). As we are focusing on games here, we need to extend the above conclusion from a single saddle problemtoasaddlegame. Let’sdefinethesaddlegameasfollows. 23 Definition4: Suppose we haveN players, and each playeri has a pair of decision variables, e.g. (x i ,y i ), with x i 2X i and y i 2Y i where X i ✓ R n i and Y i ✓ R m i are two given closed convex sets. Let n = N P i=1 n i andm = N P i=1 m i . For each playeri, the saddle functionL i : R n+m 7! R has the form ofL i (x i ,x i ,y i ,y i ) that is continuously differentiable in (x i ,y i ) N i=1 over an open set containing X⇥ Y , N Q i=1 (X i ⇥ Y i )and“convex-concave”inx i 2X i andy i 2Y i givenanyfeasible(x i ,y i ). The saddle game problem is to find a tuple (x ⇤ ,i ,y ⇤ ,i ) N i=1 2X⇥ Y named the N.E. of the saddle game,suchthatforeachplayeri,wehave: L i (x ⇤ ,i ,x ⇤ , i ,y i ,y ⇤ , i ) L i (x ⇤ ,i ,x ⇤ , i ,y ⇤ ,i ,y ⇤ , i ) L i (x i ,x ⇤ , i ,y ⇤ ,i ,y ⇤ , i ) 8(x i ,y i )2X i ⇥ Y i . Wedenotethesaddlegamedefinedaboveas: ⇢ Saddleproblemof: L i (x i ,x i ,y i ,y i ) in x i 2X i ,y i 2Y i N i=1 . (2.9) Similar to the saddle problem, the saddle game (2.9) is also equivalent to a V.I. problem. To look atthispoint,let’sdefine: F(Z), 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 r x 1L 1 (x 1 ,x 1 ,y 1 ,y 1 ) r y 1L 1 (x 1 ,x 1 ,y 1 ,y 1 ) . . . r x NL N (x N ,x N ,y N ,y N ) r y NL N (x N ,x N ,y N ,y N ) 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 withZ = 2 6 6 6 6 6 6 6 6 6 6 4 x 1 y 1 x 2 y 2 . . . x N y N 3 7 7 7 7 7 7 7 7 7 7 5 . 24 Thenwehavethefollowinglemma. Lemma3 A point (x ⇤ ,i ,y ⇤ ,i ) N i=1 2 N Q i=1 (X i ⇥ Y i ) is a N.E. of the saddle game defined in the def- inition 4 if and only if (x ⇤ ,i ,y ⇤ ,i ) N i=1 2SOL( N Q i=1 (X i ⇥ Y i ),F) where F is the mapping defined above. Proof: First, let (x ⇤ ,i ,y ⇤ ,i ) N i=1 2 N Q i=1 (X i ⇥ Y i ) be a N.E. of the saddle game defined in the definition 4. Then for each i, (x ⇤ ,i ,y ⇤ ,i ) is a saddle point of the saddle problem (L i (x i ,x ⇤ , i ,y i ,y ⇤ , i ), X i , Y i ). Using the equivalence between a saddle problem and a V.I. problem, we have (x ⇤ ,i ,y ⇤ ,i ) 2 SOL(X i ⇥ Y i ,F i )whereF i isdefinedas: F i (x i ,y i ), 0 B B @ r x iL i (x i ,x ⇤ , i ,y i ,y ⇤ , i ) r y iL i (x i ,x ⇤ , i ,y i ,y ⇤ , i ) 1 C C A . Byconcatenatingtheequivalenceresultsoveralli’s,wecanseethat (x ⇤ ,i ,y ⇤ ,i ) N i=1 2SOL( N Q i=1 (X i ⇥ Y i ),F). Reversely,if (x ⇤ ,i ,y ⇤ ,i ) N i=1 2SOL( N Q i=1 (X i ⇥ Y i ),F),wehave: (x x ⇤ ,y y ⇤ ) T · 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 r x 1L 1 (x ⇤ ,1 ,x ⇤ , 1 ,y ⇤ ,1 ,y ⇤ , 1 ) r y 1L 1 (x ⇤ ,1 ,x ⇤ , 1 ,y ⇤ ,1 ,y ⇤ , 1 ) . . . r x NL N (x ⇤ ,N ,x ⇤ , N ,y ⇤ ,N ,y ⇤ , N ) r y NL N (x ⇤ ,N ,x ⇤ , N ,y ⇤ ,N ,y ⇤ , N ) 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 0 for8(x,y), (x i ,y i ) N i=1 2 N Q i=1 (X i ⇥ Y i )where (x ⇤ ,y ⇤ ), (x ⇤ ,i ,y ⇤ ,i ) N i=1 . 25 Intheaboveinequality,foreachi,let (x i ,y i )=(x ⇤ , i ,y ⇤ , i ). Thenwecandeducethat: (x i x ⇤ ,i ,y i y ⇤ ,i ) T 0 B B @ r x iL i (x ⇤ ,i ,x ⇤ , i ,y ⇤ ,i ,y ⇤ , i ) r y iL i (x ⇤ ,i ,x ⇤ , i ,y ⇤ ,i ,y ⇤ , i ) 1 C C A 0 8(x i ,y i )2X i ⇥ Y i . Again,usingtheequivalencebetweenasaddleproblemandaV.I.problem,wehavethat(x ⇤ ,i ,y ⇤ ,i ) isthesaddlepointofthesaddleproblem (L i (x i ,x ⇤ , i ,y i ,y ⇤ , i ),X i ,Y i ). Hence, (x ⇤ ,i ,y ⇤ ,i ) N i=1 isa N.E.ofthesaddlegamedefinedinthedefinition4. ⇤ Now, we have established the equivalence between a saddle game and a V.I. problem. Back to the game we are considering here, we hope the max-min game (2.6) can be equivalent to a V.I. problem. Inordertoproveit,wejustneedtoshowthatthe max-mingame(2.6)isequivalenttoa saddlegamewhichisequivalenttoaV.I.problem. Let’sconstructtheV.I.problemasacandidatetobeequivalenttogame(2.6). Suppose assumption (A1) holds. Then for each i, his/her Lagrangian function L i (x i ,x i , i ) is continuously differentiable in (x i , i ) over an open set containing X i ⇥ ⇤ i given any x i 2X i . Also, givenx i 2X i , L i (x i ,x i , i ) is “convex-concave” inx i 2X i and i 2⇤ i . What’s more, from(A1),weknowthatX i ⇥ ⇤ i isaclosedconvexsetforeachi. Usingallthesamenotationsdefinedingame(2.6),let ˜ K, N Q i=1 (X i ⇥ ⇤ i )anddefine ˜ F(Z)as: 26 ˜ F(Z), 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 r x 1L 1 (x 1 ,x 1 , 1 ) r 1L 1 (x 1 ,x 1 , 1 ) . . . r x NL N (x N ,x N , N ) r NL N (x N ,x N , N ) 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 withZ = 2 6 6 6 6 6 6 6 6 6 6 4 x 1 1 x 2 2 . . . x N N 3 7 7 7 7 7 7 7 7 7 7 5 . (2.10) Then we expect that the game (2.6) is equivalent to VI( ˜ K, ˜ F). The following lemma shows that under assumption (A1), the game (2.6) is indeed equivalent to VI( ˜ K, ˜ F) by using a saddle game astheintermediary. Lemma4 Suppose assumption (A1) holds. Then apoint (x ⇤ ,i , ⇤ ,i ) N i=1 2 N Q i=1 (X i ⇥ ⇤ i ) is aN.E. of thegame(2.6)ifandonlyif (x ⇤ ,i , ⇤ ,i ) N i=1 solvesVI( ˜ K, ˜ F). Proof: Let’sconstructthesaddlegameplayingastheintermediaryintheanalysis. ⇢ Saddleproblemof: L i (x i ,x i , i ) in x i 2X i , i 2⇤ i N i=1 . (2.11) Since the saddle game (2.11) satisfies all the conditions mentioned in the definition 4, it is equiv- alent to VI( ˜ K, ˜ F). Therefore, to prove the lemma, we just need to show that the game (2.6) is equivalenttothesaddlegame(2.11). Suppose (x ⇤ ,i , ⇤ ,i ) N i=1 2 N Q i=1 (X i ⇥ ⇤ i )isaN.E.ofthegame(2.6). Thenforeachi,wehave: ⇤ ,i 2argmax i 0 min x i 2 X i L i (x i ,x ⇤ , i , i ), ✓ i (x i ,x ⇤ , i )+( i ) T E[G i (x i ,!)] with x ⇤ ,i 2arg min x i 2 X i ✓ i (x i ,x ⇤ , i )+( ⇤ ,i ) T E[G i (x i ,!)]. Hence,itcanbeautomaticallydeducedthat L i (x ⇤ ,i ,x ⇤ , i , ⇤ ,i )L i (x i ,x ⇤ , i , ⇤ ,i )for8x i 2X i . From lemma 2, we know that x ⇤ ,i is the unique optimal solution of the following optimization 27 problem: min x i 2 Z i ✓ i (x i ,x ⇤ , i ). (2.12) Based on the definition of (x ⇤ ,i , ⇤ ,i ), we know that ⇤ ,i is the optimal dual solution of the opti- mization problem (2.12). By using the Slater’s condition implied by (A1.6) and referring to [6], we know the pair (x ⇤ ,i , ⇤ ,i ) must satisfy the KKT conditions for the problem (2.12). Hence, we have: k i P j=1 ⇤ ,i j E[G i j (x ⇤ ,i ,!)] = 0. (2.13) Since x ⇤ ,i is feasible in the i’s sub-problem in game (2.1) implying E[G i (x ⇤ ,i ,!)]  0, given x ⇤ ,i , the value of Lagrangian function L i (x ⇤ ,i ,x ⇤ , i , i )= ✓ i (x ⇤ ,i ,x ⇤ , i )+( i ) T E[G i (x ⇤ ,i ,!)] achieves its maximum in i 2⇤ i when k i P j=1 i j E[G i j (x ⇤ ,i ,!)] = 0. Thus, from (2.13) we know that ⇤ ,i 2argmax i 2 ⇤ i L i (x ⇤ ,i ,x ⇤ , i , i ), which means L i (x ⇤ ,i ,x ⇤ , i , i )L i (x ⇤ ,i ,x ⇤ , i , ⇤ ,i ) for 8 i 2⇤ i . Therefore,foreachi,givenx ⇤ , i , (x ⇤ ,i , ⇤ ,i )isthesaddlepointofthesaddleproblem: (L i (x i ,x ⇤ , i , i ),X i ,⇤ i ). Consequently,wehavethat (x ⇤ ,i , ⇤ ,i ) N i=1 2 N Q i=1 (X i ⇥ ⇤ i )isaN.E.ofthesaddlegame(2.11). Reversely, suppose (x ⇤ ,i , ⇤ ,i ) N i=1 2 N Q i=1 (X i ⇥ ⇤ i ) is a N.E. of the saddle game (2.11). Then for each i, (x ⇤ ,i , ⇤ ,i ) is a saddle point of the saddle problem: (L i (x i ,x ⇤ , i , i ),X i ,⇤ i ). Defin- ing the dual function g i ( i ;x ⇤ , i ) , min x i 2 X i L i (x i ,x ⇤ , i , i ) and by referring to P22 [12], we have ⇤ ,i 2argmax i 2 ⇤ i g i ( i ;x ⇤ , i ). Using the definition of the saddle point, we know that L i (x ⇤ ,i ,x ⇤ , i , ⇤ ,i )L i (x i ,x ⇤ , i , ⇤ ,i ) for 8x i 2X i . Hence,given ⇤ ,i ,wehavex ⇤ ,i =argmin x i 2 X i L i (x i ,x ⇤ , i , ⇤ ,i ). 28 Astheresult,foreachplayeri, (x ⇤ ,i , ⇤ ,i )solvestheproblem max i 2 ⇤ i min x i 2 X i L i (x i ,x ⇤ , i , i ). Thus, (x ⇤ ,i , ⇤ ,i ) N i=1 isaN.E.ofthegame(2.6). ⇤ Thelemma2andlemma4togethertellusthatunderassumptions(A1)and(A2),togettheunique N.E. of the original game (2.1), we just need to solve VI( ˜ K, ˜ F) where both ˜ K and ˜ F have been definedabove. Inthenextsub-section,wewillgivetheoutlineoftheouter-loopintheLagrangianschemethatis developedtosolveVI( ˜ K, ˜ F). 2.3.3 Outer-LoopOutline Inthelastsub-section,wehaveshownthatthe max-mingame(2.6)isequivalenttoVI( ˜ K, ˜ F)and hence,inthefollowinganalysis,wejustneedtostudyonhowtosolvethisV.I.problem. In fact, it turns out that solving VI( ˜ K, ˜ F) directly is not trival. The main reason is the lack of similar DD condition on the variables ( i ) N i=1 . One way to address this difficulty is to strongly convexify i in each player i’s Lagrangian function L i (x i ,x i , i ). Fortunately, the assumptions (A1)and(A3)implythecontinuityandmonotonicityof ˜ F respectively. Andbythetheorem12.3.9 in [12], we know as long as ˜ K is a closed convex set, which is implied by (A1) and SOL( ˜ K, ˜ F) is non-empty, which is implied by the lemmas 1, 2 and 4, the VI( ˜ K, ˜ F) can be solved by using the proximal point (P.P.) method proposed in P1135 of [12]. From [12] we can see that the P.P. method is an iterative algorithm where in each iteration we actually solve a V.I. problem with the strongly monotone mapping 8 . In fact, the iterations of the P.P. algorithm form the outer-loop in 8 Similar to the definition of “monotone", a vector-valued mappingF : K✓ R n 7! R n is strongly monotone if there existsapositivevalue↵ suchthat (X Y) T (F(X) F(Y)) ↵ (X Y) T (X Y)foranyX2K andY 2K. 29 ourLagrangianscheme. Formally,wehavethefollowingdescriptionfortheouter-loop. At the outer iteration marked as v, given a point ˜ z v 1 , (˜ x v 1,i , ˜ v 1,i ) N i=1 2 ˜ K, we solve a V.I. problemVI( ˜ K, ˜ F c,˜ z v 1)withcbeingapositivevalueand ˜ F c,˜ z v 1 beingdefinedasfollows: ˜ F c,˜ z v 1(Z), 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 r x 1L 1 (x 1 ,x 1 , 1 ) r 1L 1 (x 1 ,x 1 , 1 ) . . . r x NL N (x N ,x N , N ) r NL N (x N ,x N , N ) 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 +c 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 x 1 ˜ x v 1,1 1 ˜ v 1,1 . . . x N ˜ x v 1,N N ˜ v 1,N 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 with Z = 2 6 6 6 6 6 6 6 6 6 6 4 x 1 1 x 2 2 . . . x N N 3 7 7 7 7 7 7 7 7 7 7 5 . (2.14v) Fromtheabovedefinition,wecanseethat: ˜ F c,˜ z v 1(Z)= ˜ F(Z)+c·(Z ˜ z v 1 ). Assumption (A3) implies the monotonicity of ˜ F(Z) on ˜ K and hence further implies the strong monotonicity of ˜ F c,˜ z v 1(Z) on ˜ K. Therefore, by referring to the theorem 2.3.3 in [12], we know thatforeachv,SOL( ˜ K, ˜ F c,˜ z v 1)isnon-emptyandonlycontainsonepoint,whichmeans VI( ˜ K, ˜ F c,˜ z v 1)hasauniquesolution,e.g. (x ⇤ ,i , ⇤ ,i ) N i=1 . Let(˜ x v,i , ˜ v,i ) N i=1 =(x ⇤ ,i , ⇤ ,i ) N i=1 . Wecan usethepoint ˜ z v , (˜ x v,i , ˜ v,i ) N i=1 astheparameterinthe (v+1)-thouteriteration. Hence, in ourLagrangianscheme, fixingthepositivevaluec, ineachouteriterationv, weneedto solve VI( ˜ K, ˜ F c,˜ z v 1) and then use the solution denoted by ˜ z v 1 as the parameter used in the next outeriteration. Ineachouteriterationv,giventhestrongmonotonicityofthemapping ˜ F c,˜ z v 1,wecansuccessfully solvetheV.I.problemVI( ˜ K, ˜ F c,˜ z v 1)bydevelopinganinner-loop. Beforeweintroducetheinner-looptosolvethestronglymonotoneV.I.problem,e.g. VI( ˜ K, ˜ F c,˜ z v 1), 30 ineachouteriterationv,let’sshowthattheVI( ˜ K, ˜ F c,˜ z v 1)isactuallyequivalenttoanewmax-min (or min-max) game, which will be helpful in the practical implementations in the last chapter of thethesis. Formally,foreachouteriterationv,define ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ), ✓ i (x i ,x i )+( i ) T E[G i (x i ,!)]+ c 2 kx i ˜ x v 1,i k 2 c 2 k i ˜ v 1,i k 2 foreachi. From the definition, we have ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i )= L i (x i ,x i , i )+ c 2 kx i ˜ x v 1,i k 2 c 2 k i ˜ v 1,i k 2 . Let’scallthefunction ˜ L r i asthe“RegularizedLagrangianFunction”ofplayeri. Definea max-mingameasbelow: ⇢ max i 0 min x i 2 X i ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ) N i=1 . (2.15v) Andsimilarly,definea min-maxgameasfollows: ⇢ min x i 2 X i max i 0 ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ) N i=1 . (2.15 0 v) The definition of the N.E. in the game (2.15v) (or (2.15 0 v)) is similar to the corresponding defini- tionoftheN.E.inthegame(2.6). In (2.15v), since given x i 2X i and i 2⇤ i , ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ) is strongly convex in x i 2X i , which means the minimizer of the function ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ) in x i 2X i is obtainable, the use of min, instead of inf, in the definition of (2.15v) is meaningful. Define ˜ g v i ( i ;x i ), min x i 2 X i ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ). Then it is straightforward to see that ˜ g v i ( i ;x i ) is strongly concave in i 2⇤ i . Hence, the use of max, instead of sup, in the definition of (2.15v) isalsomeaningfulduetotheexistenceofthemaximizerofthefunction ˜ g v i ( i ;x i )in i 2⇤ i . 31 Similarly,inthedefinitionof (2.15 0 v),theusesof minand max,insteadof inf and suprespective, arealsomeaningful. Now, we can prove the following lemma to establish the equivalence between VI( ˜ K, ˜ F c,˜ z v 1) and the max-mingame(2.15v)(orthe min-maxgame(2.15 0 v)). Lemma5 Supposeassumptions(A1)and(A3)hold. Thenatanyouteriterationv,givenapositive constantc> 0andapoint(˜ x v 1,i , ˜ v 1,i ) N i=1 2 ˜ K,apoint(x ⇤ ,i , ⇤ ,i ) N i=1 2 ˜ K solvesVI( ˜ K, ˜ F c,˜ z v 1) ifandonlyifitisaN.E.ofthegame(2.15v)(or(2.15 0 v)). Proof: Focusononefixedouteriterationv. Supposeapositiveconstantcandapoint(˜ x v 1,i , ˜ v 1,i ) N i=1 2 ˜ K aregivenatv. First,let’sconstructthesaddlegameusedinthisanalysisasbelow: ⇢ Saddleproblemof: ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ) in x i 2X i , i 2⇤ i N i=1 . (2.16) Thenfromdefinition4andlemma3,weknowthatthesaddlegame(2.16)isequivalentto VI( ˜ K, ˜ F c,˜ z v 1). Hence, in this proof, we just need show the equivalence between the saddle game (2.16)andthe max-mingame(2.15v)(orthe min-maxgame(2.15 0 v)). As assumptions (A1) and (A3) together imply that ˜ F c,˜ z v 1 is continuous and strongly monotone on ˜ K and ˜ K is a closed convex set, by referring to the theorem 2.3.3 in [12], we know that VI( ˜ K, ˜ F c,˜ z v 1) has a unique solution. As a result, the N.E. of the saddle game (2.16) exists and is unique. Suppose (x ⇤ ,i , ⇤ ,i ) N i=1 2 ˜ K is the unique N.E. of the saddle game (2.16). Then for each player 32 i, (x ⇤ ,i , ⇤ ,i )isasaddlepointofthesaddleproblem ( ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ),X i ,⇤ i ),which means: ˜ L r i (x ⇤ ,i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ) ˜ L r i (x ⇤ ,i ,x ⇤ , i , ⇤ ,i ;˜ x v 1,i , ˜ v 1,i )  ˜ L r i (x i ,x ⇤ , i , ⇤ ,i ;˜ x v 1,i , ˜ v 1,i ) 8(x i , i )2X i ⇥ ⇤ i . (2.17) ThenbyP22in[12],wehave ⇤ ,i =argmax i 2 ⇤ i ✓ ˜ g v i ( i ;x ⇤ , i ), min x i 2 X i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ) ◆ . Combiningwiththesecondinequalityin(2.17),wegetthat: ⇤ ,i =argmax i 2 ⇤ i min x i 2 X i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ) withx ⇤ ,i =argmin x i 2 X i ˜ L r i (x i ,x ⇤ , i , ⇤ ,i ;˜ x v 1,i , ˜ v 1,i ). Reversely,let (x ⇤ ,i , ⇤ ,i ) N i=1 2 ˜ K beaN.E.ofthegame (2.15v). Thenforeachi,wehave: ⇤ ,i =argmax i 2 ⇤ i min x i 2 X i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ) withx ⇤ ,i =argmin x i 2 X i ˜ L r i (x i ,x ⇤ , i , ⇤ ,i ;˜ x v 1,i , ˜ v 1,i ). Now,weneedtoshowthat (x ⇤ ,i , ⇤ ,i )isasaddlepointofthesaddleproblem: ( ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ),X i ,⇤ i ). By using the assumption (A1) and recalling the respective minimum and maximum principles for the two problems: min x i 2 X i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ) and max i 2 ⇤ i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ), we havethat(¯ x i , ¯ i )2X i ⇥ ⇤ i isasaddlepointoftheproblem( ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ),X i ,⇤ i ) ifandonlyif(¯ x i , ¯ i )solvestheVI(X i ⇥ ⇤ i , ˜ F i )where 33 ˜ F i (x i , i ), 0 B B @ r x i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ) r i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ) 1 C C A , (x i , i )2X i ⇥ ⇤ i . Itcanbeeasilyverifiedthatthemapping ˜ F i (x i , i )definedaboveiscontinuousandstronglymono- toneinX i ⇥ ⇤ i whichisaclosedconvexset. Hence,theVI(X i ⇥ ⇤ i , ˜ F i )hasauniquesolutionand therefore,thesaddleproblem( ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ),X i ,⇤ i )hasthesameuniquesolution, e.g. (¯ x i , ¯ i ). Usingthesimilaranalysisinthefirstpartoftheproofinthislemma,weknowthat: ¯ i 2argmax i 2 ⇤ i min x i 2 X i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ) with ¯ x i 2arg min x i 2 X i ˜ L r i (x i ,x ⇤ , i , ¯ i ;˜ x v 1,i , ˜ v 1,i ). Meanwhile,in i 2⇤ i ,thefunction ˜ g v i ( i ;x ⇤ , i )= min x i 2 X i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i )isstrongly concavein i andgivenany i 2⇤ i ,thefunction ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i )isstronglyconvex in x i 2X i . Then the max-min problem: max i 2 ⇤ i min x i 2 X i ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ) has a unique solutionpair. However,wehaveshownthatboth(x ⇤ ,i , ⇤ ,i )and(¯ x i , ¯ i )solvetheabovemax-minproblem. Thus, wehave (x ⇤ ,i , ⇤ ,i )=(¯ x i , ¯ i ),whichmeans (x ⇤ ,i , ⇤ ,i )isasaddlepointofthesaddleproblem ( ˜ L r i (x i ,x ⇤ , i , i ;˜ x v 1,i , ˜ v 1,i ),X i ,⇤ i ). Hence, we can establish the equivalence between the VI( ˜ K, ˜ F c,˜ z v 1) and the max-min game (2.15v). And under almost the same logics, we can easily establish the equivalence between the VI( ˜ K, ˜ F c,˜ z v 1)andthe min-maxgame (2.15 0 v)aswell. ⇤ Thelemma5actuallygivesustheequivalencesoverallthethreeproblemsincludingthe 34 VI( ˜ K, ˜ F c,˜ z v 1),the max-mingame (2.15v)andthe min-maxgame (2.15 0 v). Now,byusingtheconclusionoflemma5,weknowthatineachouteriterationv inourLagrangian scheme, we just need to solve the max-min game (2.15v) (or a min-max game (2.15 0 v)). As the result, we develop an iterative method to solve (2.15v) (or (2.15 0 v)) whose iterations form the inner-loopoftheLagrangianscheme. 2.3.4 Inner-LoopOutline In this sub-section, we will study on how to solve the max-min (or min-max) game (2.15v) (or (2.15 0 v)) in the inner-loop under each outer iteration v. We are noting that the game (2.15v) (or (2.15 0 v)) is equivalent to VI( ˜ K, ˜ F c,˜ z v 1). Essentially, VI( ˜ K, ˜ F c,˜ z v 1) is a Stochastic Variational Inequality (SVI) problem [45] with the mapping ˜ F c,˜ z v 1 being strongly monotone on ˜ K. There are already some studies of SVI problems [21,37,40,45,46]. Especially, in [21], the authors have shown that under some mild assumptions, the SVI problem with the strongly monotone mapping canbesolvedbyapplyingtheStochasticApproximation(SA)approach. Theauthorsproposedan iterativemethodwhereineachiteration,peopleneedtosolveaprojectionproblem. It turns out that the V.I problem VI( ˜ K, ˜ F c,˜ z v 1) considered here can be solved by using the SA approach mentioned above. However, the number of iterations in the SA approach may be very large, which means we may need to solve so many projection problems and each one is actually an optimization problem. Thus, the time efficiency of the SA approach may be very low. Most importantly, in this thesis, we are studying on solving the stochastic games. Nevertheless, the projection method cannot embody the spirit of game. In a game, for each player, given other players’strategies,he/sheneedstosolvehis/hersub-problemtogetthebest-responsesolution. The projectionmethodisdefinitelynotabletopresentsuchprincipal. Hence,inthisthesis,undereach 35 outer iterationv, we will develop an inner-loop combining the “Sample Average Approximation” (SAA)method[43]andthe“Best-Response”(BR)algorithem[13]. Specifically,tosolvethegame (2.15v): ⇢ max i 0 min x i 2 X i ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ) N i=1 where ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i )= ✓ i (x i ,x i )+( i ) T E[G i (x i ,!)]+ c 2 kx i ˜ x v 1,i k 2 c 2 k i ˜ v 1,i k 2 foreachi,wehavetheinner-loopdescribedasbelow. Starting from µ=0 with (x 0,v,i , 0,v,i )=(˜ x v 1,i , ˜ v 1,i ) 2X i ⇥ ⇤ i for each player i, we ob- tain (x µ+1,v,i , µ+1,v,i )foreachiusingthefollowingrule: µ+1,v,i = argmax i 0 min x i 2 X i 1 L v µ L v µ P s=1 H i (x i ,x µ,v, i ,! s )+( i ) T 1 L v µ L v µ P s=1 G i (x i ,! s ) + c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 with x µ+1,v,i =argmin x i 2 X i 1 L v µ L v µ P s=1 H i (x i ,x µ,v, i ,! s )+( µ+1,v,i ) T 1 L v µ L v µ P s=1 G i (x i ,! s ) + c 2 ||x i ˜ x v 1,i || 2 c 2 || µ+1,v,i ˜ v 1,i || 2 . (2.18) Here, L v µ is the number of samples used in iterationµ and will go to1 asµ goes to1. (! s ) L v µ s=1 representsthegeneratedindependentlyandidenticallydistributed(i.i.d.) randomsamples. Oncewehaveobtained (x µ+1,v,i , µ+1,v,i ) N i=1 ,setµ µ+1anditerativelyobtainthepointof (x µ+1,v,i , µ+1,v,i ) N i=1 inthenextinneriteration. 36 Let’sdefine: ˜ L sr i,µ (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ), 1 L v µ L v µ X s=1 H i (x i ,x i ,! s )+( i ) T 1 L v µ L v µ X s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 and call it as the “Sample Regularized Lagrangian Function” of player i on the inner iteration µ underouteriterationv. Inthe max-minproblem: argmax i 0 min x i 2 X i 1 L v µ L v µ P s=1 H i (x i ,x µ,v, i ,! s )+( i ) T 1 L v µ L v µ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 , suppose assumption (A1) holds. Then, for each player i, like what we have shown in the proof of lemma 5, we can deduce the similar strong convexity and strong concavity properties for the corresponding functions respectively. And hence, (x µ+1,v,i , µ+1,v,i ) is the unique solution pair of theabove max-minproblemforeachµandv. In the next section, we will prove that under assumptions (A1) to (A3), in each outer iteration v, the inner-loop solutions sequence {(x µ,v,i , µ,v,i ) N i=1 } 1 µ=1 will converge to the unique N.E. of the game (2.15v)inalmosteverywhere(a.e.) sense. 2.3.5 DescriptionoftheLagrangianScheme In this sub-section, let’s first summarize the conclusions we have made so far and then give the Lagrangianscheme’sdescriptionindetail. Supposealltheassumptions(A1)-(A3)hold. First,inlemma1,wehaveshowntheexistenceand uniqueness of the N.E. in original game (2.1). Next, in lemma 2, we established the equivalence between the original game (2.1) and the max-min game (2.6). In lemma 4, we showed that the 37 max-min game (2.6) is equivalent to a V.I. problem whose mapping ˜ F is defined in (2.10). Then weintroducedtheP.P.algorithmtoinducetheouter-loopofourLagrangianscheme. Andalso,we gave the proof to illustrate that the V.I. problem in each outer iteration v whose mapping ˜ F c,˜ z v 1 is defined in (2.14v) is equivalent to another max-min (or min-max) game defined in (2.15v) (or (2.15 0 v)). Hence, in each outer iteration v of the Lagrangian scheme, we actually solve the max-min game (2.15v). Lastly, to solve (2.15v), we developed the inner-loop in our Lagrangian scheme. Theupdatingmethodofthesolutionineachinneriterationispresentedin(2.18). Thefollowingrelations’chainsummarizesthelogicsofouranalysesinthissectionsofar. Originalgame(2.1), max-mingame(2.6), VI( ˜ K, ˜ F) P.P. ( VI( ˜ K, ˜ F c,˜ z v 1), max-mingame (2.15v)( Inner-loopdefinedin(2.18). All the double-headed arrows in the above chain have been verified in this section. The only thingremainedistoshowwhetherthetworelationsimpliedbythetwoleftarrowsrespectivelyare correct. In the next section, we will give the proof of the first left arrow, e.g. P.P. (, in the outer-loop convergence analysis. And then, in the section of the inner-loop convergence analysis, the second leftarrowwillbeverified. Astheconclusionofthissection,let’sgivetheoveralldescriptionoftheLagrangianscheme. Algorithm1: LagrangianScheme Undereachouteriterationv,let(L v µ ) 1 µ=0 beapositivenon-decreasingsequencewith lim µ!1 L v µ =1. 38 Foreachplayeri,define ˜ H i µ,v (x i ,x i ), 1 L v µ L v µ P s=1 H i (x i ,x i ,! s )foreachµandv andanysequence ofi.i.d. randomsamples (! s ) L v µ s=1 . Fix a positive value c. And arbitrarily select an initial point (˜ x 0,i , ˜ 0,i ) N i=1 with ˜ x 0,i 2X i and ˜ 0,i 2⇤ i fori=1,...,N. Startingfromthefirstouteriterationv=1,wehavethefollowingsteps. (1)Setµ=0andlet (x 0,v,i , 0,v,i ) N i=1 =(˜ x v 1,i , ˜ v 1,i ) N i=1 . (2) Newly generate the L µ,v i.i.d. random samples of ! where L µ,v = L v 0 if µ=0 or L µ,v = L v µ L v µ 1 otherwise. Thenintheouteriterationv,wehavealreadygeneratedtheL v µ i.i.d. random samplesof!,e.g. (! s ) L v µ s=1 . (3)Fori=1,...,N,define (x µ+1,v,i , µ+1,v,i )asfollows: µ+1,v,i , argmax i 0 min x i 2 X i ˜ H i µ,v (x i ,x µ,v, i )+( i ) T 1 L v µ L v µ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 with x µ+1,v,i , arg min x i 2 X i ˜ H i µ,v (x i ,x µ,v, i )+( µ+1,v,i ) T 1 L v µ L v µ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || µ+1,v,i ˜ v 1,i || 2 . (4) If some criterion is satisfied, we proceed to step (5). Otherwise, updateµ µ+1 and return to(2). (5)Fori=1,...,N,assign ˜ x v,i =x µ+1,v,i and ˜ v,i = µ+1,v,i . (6) If some criterion is satisfied, we terminate this algorithm and take (˜ x v,i ) N i=1 as the computed N.E.ofthegame(2.1). Otherwise,updatev v+1andreturnto(1). We have provided the big picture of the Lagrangian scheme. Next, we will investigate whether in thisalgorithm,bothouterandinner-loopsconvergetothedesirablepointsrespectively. 39 2.4 Outer-LoopConvergenceAnalysis In the last section, we have described the Lagrangian scheme. In the description, we can see that there are two loops. In each outer iterationv, we need to solve the max-min game (2.15v). And ineachinner-loop,wecombinetheSAAandBRtosolveit. In this section, we will give the proof for outer-loop convergence property. That is, the outer-loop solutionssequence,e.g. (˜ x v , ˜ v ) 1 v=1 ,willconvergetoonesolutionofVI( ˜ K, ˜ F). Likewhatwehave highlighted, in the outer-loop convergence analysis, we are actually verifying the relation implied by P.P. (intherelationschaingivenin2.3.5. Now, let’s start our arguments. In the last section, we have shown that to solve the original game (2.1),wejustneedtosolvetheV.I.problemVI( ˜ K, ˜ F). AndtosolvethisV.I.problem,wehavede- velopedtheouter-loopinourLagrangianscheme. Thatisineachouteriterationv,given ˜ z v 1 2 ˜ K, wesolvetheVI( ˜ K, ˜ F c,˜ z v 1)withcbeingafixedpositivevalue. Wehaveshownthatforeachouter iterationv, the VI( ˜ K, ˜ F c,˜ z v 1) has a unique solution provided that the assumptions (A1) and (A3) hold. Welet ˜ z v denotethisuniquesolutionandthenuseitastheparametertosolvetheVI( ˜ K, ˜ F c,˜ z v) inthe (v+1)-thouteriteration. From the definitions of ˜ F and ˜ F c,˜ z v 1, we have ˜ F c,˜ z v 1(z)= ˜ F(z)+c· (z ˜ z v 1 ) for allz2 ˜ K. Hence, ˜ z v istheuniquesolutionofVI( ˜ K, ˜ F c,˜ z v 1)ifandonlyif: (z ˜ z v ) T ( ˜ F(˜ z v )+c·(˜ z v ˜ z v 1 )) 0, 8z2 ˜ K. Therefore, due to the closeness of ˜ K and the continuity of ˜ F on ˜ K, if the solutions sequence (˜ z v ) 1 v=1 converges,e.g. lim v!1 ˜ z v =˜ z 1 , ˜ z 1 mustbeonesolutionofVI( ˜ K, ˜ F). Theouter-loopintheLagrangianschemeisactuallyanimplementationoftheP.P.algorithmmen- tioned in the last section. Various kinds of convergence properties of this algorithm have been 40 well investigated in [12] . By referring to the theorem 12.3.9 in [12] , formally we can prove the followingtheorem. Theorem1 Supposetheassumptions(A1)-(A3)hold. Thenforanypositivecandtheinitialpoint (˜ x 0,i , ˜ 0,i ) N i=1 2 ˜ K, the induced solutions sequence in the outer-loop of the Lagrangian scheme, e.g. ((˜ x v,i , ˜ v,i ) N i=1 ) 1 v=1 ,willconvergetoonesolutionofVI( ˜ K, ˜ F). Proof: Assumption(A1)impliesthenon-emptiness,closenessandconvexityof ˜ K. Also,under(A1)and (A3), ˜ F is continuous and monotone on ˜ K. What’s more, we should note that for any positive value c and in any outer iteration v, ˜ z v 2SOL( ˜ K, ˜ F c,˜ z v 1) if and only if the following condition holds: (z ˜ z v ) T ( 1 c ˜ F(˜ z v )+(˜ z v ˜ z v 1 )) 0, 8z2 ˜ K. Furthermore, lemma1showsthatunderassumptions(A1)and(A2), theoriginalgame(2.1)hasa uniqueN.E.,whichmeansSOL( ˜ K, ˜ F)isnon-emptybytheconclusionsoflemma2andlemma4. Hence, by referring to the proposition 12.3.6 and the theorem 12.3.9 in [12] and noting that the outer-loopinourLagrangianschemeisanimplementationoftheP.P.algorithmdescribedintheo- rem12.3.9withc v = 1 c ,✏ v =0and⇢ v =1forallv’s, wecandeducethattheouter-loopsolutions sequenceinourLagrangianschemeconvergestoapointinSOL( ˜ K, ˜ F). ⇤ Now,wehaveestablishedtheconvergenceresultfortheouter-loopintheLagrangianscheme. The only thing remained in this chapter is to prove that in each outer iterationv, in some probabilistic sense, the solutions sequence in the corresponding inner-loop will converge to ˜ z v =(˜ x v,i , ˜ v,i ) N i=1 41 whichistheuniqueN.E.solutionofthe max-mingame (2.15v). 2.5 Inner-LoopConvergenceAnalysis When we check the updating scheme shown in (2.18), we can see that for each outer iteration v, thecorrespondinginner-loopsolutionssequence ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 israndomwhosevaluede- pends on both the point (˜ x v 1 , ˜ v 1 ) and the specific realization of the random sequence (! s ) 1 s=1 . Hence, we need to investigate the inner-loop convergence property under a specific probabilistic sense. We will show that under assumptions (A1) to (A3), ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 will converge to theuniqueN.E.ofthegame (2.15v)inalmosteverywhere(a.e.) sense. Now,let’srecaptheupdatingschemeintheinner-loopiterations. Starting from µ =0 with (x 0,v,i , 0,v,i ) N i=1 =(˜ x v 1,i , ˜ v 1,i ) N i=1 2 ˜ K, for each i, we obtain (x µ+1,v,i , µ+1,v,i )usingthefollowingrule: µ+1,v,i = argmax i 0 min x i 2 X i 1 L v µ L v µ P s=1 H i (x i ,x µ,v, i ,! s )+( i ) T 1 L v µ L v µ P s=1 G i (x i ,! s ) + c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 with x µ+1,v,i =argmin x i 2 X i 1 L v µ L v µ P s=1 H i (x i ,x µ,v, i ,! s )+( µ+1,v,i ) T 1 L v µ L v µ P s=1 G i (x i ,! s ) + c 2 ||x i ˜ x v 1,i || 2 c 2 || µ+1,v,i ˜ v 1,i || 2 . (2.18) Insection2.3,wehavediscussedtheexistenceanduniquenessoftheabove max-minproblem. Ineachouteriterationv,toensurethatthesequenceof ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 convergesoratleast, hassomeaccumulationpoints,weneedtosetupsomeboundsonbothx i and i foreachplayeri. Due to the compactness ofX i , we only need to give the bound for the i part. Since in each outer 42 iterationv,weonlyrestrict i tobeanon-negativevector,theboundednessof i cannotbesatisfied automatically. Hence,inthissection,weneedtoprovetheboundednessof (( µ,v,i ) N i=1 ) 1 µ=1 atfirst. Then we can show the convergence of the inner-loop solutions sequence using its boundedness property. 2.5.1 Boundednessofthe Part It turns out that if the Slater’s condition holds for each playeri’s expected-value constraints in the originalgame(2.1),thenintheLagrangianscheme,undereachouteriterationsv,the partofthe inner-loop solutions sequence, e.g. (( µ,v,i ) N i=1 ) 1 µ=1 , will be bounded in almost everywhere (a.e.) sense. From assumption (A1.5), we know with probability 1, the realized random vectors sequence w, (! s ) 1 s=1 makesthefollowingfourequalitiesholdforallplayeri’s: lim L!1 [sup x2 X || 1 L L P s=1 H i (x i ,x i ,! s ) E[H i (x i ,x i ,!)]|| 2 ]=0, (2.19) lim L!1 [sup x2 X || 1 L L P s=1 r x iH i (x i ,x i ,! s )r x iE[H i (x i ,x i ,!)]|| 2 ]=0, (2.20) lim L!1 [sup x i 2 X i || 1 L L P s=1 G i (x i ,! s ) E[G i (x i ,!)]|| 2 ]=0, (2.21) lim L!1 [sup x i 2 X i || 1 L L P s=1 r x iG i (x i ,! s )r x iE[G i (x i ,!)]|| 2 ]=0. (2.22) Withthoseequalities,let’sshowtheboundednessofthe partinthefollowinglemma. Lemma6 Suppose the assumption (A1) is satisfied. Then there exist positive values B and M such that under any outer iterationv and the sequnecnew, (! s ) 1 s=1 making equalities (2.19) to (2.22) hold, there exists a positive integer ¯ L w such that at any inner iteration µ with L v µ ¯ L w , 43 ( µ,v,i ) N i=1 isboundedas k i P j=1 µ,v,i j  B +M·k ˜ v 1 k 2 forallplayers (i=1,2,3,...,N). Andthus wecansaythatinanyouteriterationv,ina.e.,thesequence( µ,v,i ) 1 µ=1 willbeboundeduniformly overallplayerseventually. Proof: From now, we restrict our analysis on the outer iteration v. Also, let’s fix one random vectors sequence, e.g. w , (! s ) 1 s=1 , making the equalities (2.19) to (2.22) hold for all i’s. Since the sequence (! s ) 1 s=1 hasbeenfixednow, ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 isdeterministicastheresult. Basedonassumption(A1),weknowtheSlater’sconditionholdsforeachplayeri’sexpected-value constraints in the original game (2.1). Let ˆ x i be the Slater’s point of playeri. Then for eachi, we haveE[G i (ˆ x i ,!)]< 0. Builtonthesettingofw,wecandeducethatthereexistsL w > 0suchthat foranyL v µ L w , 1 L v µ L v µ P s=1 G i j (ˆ x i ,! s )< 1 2 E[G i j (ˆ x i ,!)]< 0foralli’sandj’s. For eachi, define ⇣ i , max j=1,2,3,...,k i 1 2 E[G i j (ˆ x i ,!)]. Hence ⇣ i < 0 and 1 L v µ L v µ P s=1 G i j (ˆ x i ,! s )<⇣ i for all j’s as long asL v µ L w . Taking ⇣=max i=1,2,3,...,N ⇣ i , whenL v µ L w , we have 1 L v µ L v µ P s=1 G i j (ˆ x i ,! s )<⇣ foralli’sandj’s. Inordertomakethestatementsmoreclear,foreachinneriterationµandplayeri,were-writethe updatingschemeof (x µ+1,v,i , µ+1,v,i )shownin(2.18)asbelow: µ+1,v,i = argmax i 0 min x i 2 X i ˜ H i µ,v (x i ,x µ,v, i )+( i ) T 1 L v µ L v µ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 with x µ+1,v,i = arg min x i 2 X i ˜ H i µ,v (x i ,x µ,v, i )+( µ+1,v,i ) T 1 L v µ L v µ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || µ+1,v,i ˜ v 1,i || 2 where ˜ H i µ,v (x i ,x i ), 1 L v µ L v µ P s=1 H i (x i ,x i ,! s ). In the proof of lemma 5, we have asserted that the max-min game (2.15v) is equivalent to a 44 V.I. problem, e.g. VI( ˜ K, ˜ F c,˜ z v 1). And the max-min problem presented by (2.18) above actu- ally can be treated as a single player’s max-min game with ✓ i (x i ,x i ) and E[G i (x i ,!)] being replaced by ˜ H i µ,v (x i ,x µ,v, i ) and 1 L v µ L v µ P s=1 G i (x i ,! s ) respectively. By assumption (A1), we know thatboththepropertiesofconvexity(strongconvexity)anddifferentiabilityareremainedvalidfor ˜ H i µ,v (x i ,x µ,v, i )and 1 L v µ L v µ P s=1 G i (x i ,! s )correspondingly. Hence, using the similar deductions, we have that the above max-min problem in (2.18) is equiv- alenttotheV.I.problemVI(X i ⇥ ⇤ i , ˜ F µ,v,i c,˜ z v 1 )with ˜ F µ,v,i c,˜ z v 1 beingdefinedasbelow: ˜ F µ,v,i c,˜ z v 1 , 0 B B @ r x iL sr i,µ (x i ,x µ,v, i , i ;˜ x v 1,i , ˜ v 1,i ) r iL sr i,µ (x i ,x µ,v, i , i ;˜ x v 1,i , ˜ v 1,i ) 1 C C A where L sr i,µ (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i )= 1 L v µ L v µ P s=1 H i (x i ,x i ,! s )+( i ) T 1 L v µ L v µ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 whichhasbeendefinedinthelastsection. Basedontheaboveanalysis,wehave: (x i x µ+1,v,i ) T (r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )+ k i X j=1 µ+1,v,i j 1 L v µ L v µ X s=1 r x iG i j (x µ+1,v,i ,! s )) +c·(x i x µ+1,v,i ) T (x µ+1,v,i ˜ x v 1,i ) ( i µ+1,v,i ) T 1 L v µ L v µ X s=1 G i (x µ+1,v,i ,! s ) +c·( i µ+1,v,i ) T ( µ+1,v,i ˜ v 1,i ) 0 8(x i , i )2X i ⇥ ⇤ i . (2.23) Intheaboveinequality,let i = µ+1,v,i . Thenwehave: 45 (x i x µ+1,v,i ) T (r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )+ k i X j=1 µ+1,v,i j 1 L v µ L v µ X s=1 r x iG i j (x µ+1,v,i ,! s ))+ c·(x i x µ+1,v,i ) T (x µ+1,v,i ˜ x v 1,i ) 0 8x i 2X i . (2.24) In(2.24),replacingx i bytheSlater’spoint ˆ x i ,wecanreachthefollowinginequality: (ˆ x i x µ+1,v,i ) T (r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )+ k i X j=1 µ+1,v,i j 1 L v µ L v µ X s=1 r x iG i j (x µ+1,v,i ,! s ))+ c·(ˆ x i x µ+1,v,i ) T (x µ+1,v,i ˜ x v 1,i ) 0. (2.25) Similarly,in(2.23),ifwesetx i tobex µ+1,v,i ,wecanhave: ( i µ+1,v,i ) T ( 1 L v µ L v µ X s=1 G i (x µ+1,v,i ,! s ) c·( µ+1,v,i ˜ v 1,i )) 0 8 i 0. (2.26) Frominequality(2.26),wehave: 0 µ+1,v,i ? 1 L v µ L v µ X s=1 G i (x µ+1,v,i ,! s )+c·( µ+1,v,i ˜ v 1,i ) 0 ) µ+1,v,i j ·( 1 L v µ L v µ X s=1 G i j (x µ+1,v,i ,! s )+c·( µ+1,v,i j ˜ v 1,i j )) = 0 8j=1,2,3,...,k i ) µ+1,v,i j ·( 1 L v µ L v µ X s=1 G i j (x µ+1,v,i ,! s ))+c·( µ+1,v,i j ˜ v 1,i j ) 2 +c· ˜ v 1,i j ·( µ+1,v,i j ˜ v 1,i j )=0 8j ) 0 µ+1,v,i j ·( 1 L v µ L v µ X s=1 G i j (x µ+1,v,i ,! s )) c·( ˜ v 1,i j ) 2 8j. (2.27) BasedontheconvexityofG i j (x i ,!)inx i 2X i underanyrealizationof! andthefactthat µ,v,i j is 46 non-negativeforanyi,j,µ,wehave: (ˆ x i x µ+1,v,i ) T ( k i P j=1 µ+1,v,i j 1 L v µ L v µ P s=1 r x iG i j (x µ+1,v,i ,! s ))  k i P j=1 µ+1,v,i j · 1 L v µ L v µ P s=1 (G i j (ˆ x i ,! s ) G i j (x µ+1,v,i ,! s )). (2.28) Combining(2.25)and(2.28),wecandeducethat: (ˆ x i x µ+1,v,i ) T r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )+ k i X j=1 µ+1,v,i j · 1 L v µ L v µ X s=1 (G i j (ˆ x i ,! s ) G i j (x µ+1,v,i ,! s )) +c·(ˆ x i x µ+1,v,i ) T (x µ+1,v,i ˜ x v 1,i ) 0. (2.29) Incorporating(2.27)into(2.29),wehave: (ˆ x i x µ+1,v,i ) T r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )+c· k i X j=1 ( ˜ v 1,i j ) 2 + k i X j=1 ( µ+1,v,i j · 1 L v µ L v µ X s=1 G i j (ˆ x i ,! s )) +c·(ˆ x i x µ+1,v,i ) T (x µ+1,v,i ˜ x v 1,i ) 0. (2.30) Since we have already shown that there exists a positive integerL w such that whenL v µ L w , we have 1 L v µ L v µ P s=1 G i j (ˆ x i ,! s )<⇣< 0 for alli’s andj’s, from inequality (2.30) and due to µ+1,v,i j 0 forallj’s,wehave: 0 (ˆ x i x µ+1,v,i ) T r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i ) | {z } B µ+1,v,i 1 +c·(ˆ x i x µ+1,v,i ) T (x µ+1,v,i ˜ x v 1,i ) | {z } B µ+1,v,i 2 + c· k i P j=1 ( ˜ v 1,i j ) 2 +⇣ k i P j=1 µ+1,v,i j ) ⇣ k i P j=1 µ+1,v,i j  B µ+1,v,i 1 +B µ+1,v,i 2 +c· k i P j=1 ( ˜ v 1,i j ) 2 . Fromequality(2.20),weknowthatthereexistsL 0 w suchthat 47 kr x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i ) E[r x iH i (x µ+1,v,i ,x µ,v, i ,!)]k < 1 for allL v µ L 0 w . Also from as- sumption(A1),wehavethatkE[r x iH i (x i ,x i ,!)]k =kr x iE[H i (x i ,x i ,!)]k =kr x i✓ i (x i ,x i )k which is continuous in (x i ,x i ) on X implying thatkE[r x iH i (x i ,x i ,!)]k has an upper bound onX. Hence,wehavethatkr x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )kk E[r x iH i (x µ+1,v,i ,x µ,v, i ,!)]k+ kr x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i ) E[r x iH i (x µ+1,v,i ,x µ,v, i ,!)]kandtheRHSofthisinequalityhasan upperboundforanyµsuchthatL v µ L 0 w . Then considering the fact that (x µ+1,v,i ,x µ,v, i ) 2X for all µ’s and v’s, we can guarantee when L v µ L 0 w , both the B µ+1,v,i 1 and B µ+1,v,i 2 are bounded above by a value ˜ B over any µ, v, i and ˜ B is only dependent on the set X and the functions {✓ i (x i ,x i )} N i=1 . Let B 0 =2 ˜ B. Then, when L v µ ¯ L w , max(L w ,L 0 w ),wehave: ⇣ k i X j=1 µ+1,v,i j  B 0 +c· k i X j=1 ( ˜ v 1,i j ) 2 ) k i X j=1 µ+1,v,i j  1 ⇣ (B 0 +c· k i X j=1 ( ˜ v 1,i j ) 2 ) ) k i X j=1 µ+1,v,i j  1 ⇣ ·B 0 +( c ⇣ ·k ˜ v 1,i k 2 ) 1 ⇣ ·B 0 +( c ⇣ ·k ˜ v 1 k 2 ). (2.31) Foreachi,thelastinequalitycomesfromthefactthat ˜ v 1,i isasub-vectorof ˜ v 1 . In inequality (2.31), as both the ⇣ and B 0 are independent of w, if we define B , 1 ⇣ ·B 0 and M, c ⇣ ,wecanestablishtheconclusioninthestatementofthelemma. ⇤ Now, we have established the boundedness of the inner-loop solutions sequence in almost every- 48 where (a.e.) sense. In the next sub-section, we will give the proof of the convergence property of theinner-loopsolutionssequenceunderanyouteriterationv. 2.5.2 Inner-LoopConvergenceProof Inthelastsub-section,wehaveshownthatwithprobability1,theaccumulationpointoftheinner- loop solutions sequence exists due to the boundedness. Recalling that the game (2.15v) has a unique N.E., the next thing is to show that in each outer iterationv, any accumulation point of the inner-loop solutions sequence is the unique N.E. of the game (2.15v) in a.e.. If that is true, it is easytofurthershowthatthewholeinner-loopsolutionssequencewillconvergetothisuniqueN.E. ofthegame (2.15v)ina.e.. Now, let’s investigate the almost everywhere convergence property of the inner-loop solutions sequence. Theorem2 IntheLagrangianScheme,iftheassumptionsA(1)to(A3)hold,thenundereachouter iteration v, in almost everywhere (a.e.), the inner-loop solutions sequence {(x µ,v,i , µ,v,i ) N i=1 } 1 µ=1 willconvergetotheuniqueN.E.ofthegame (2.15v). Proof: Throughoutthewholeproof,wewillfixanouteriterationv. Byassumption(A1.5),weknowthat the probability that the generated random sequence w =(! s ) 1 s=1 makes the equalities (2.19) to (2.22)holdforalli’sis1. Hence,inthefollowinganalysis,weassumethatthoseequalitiesholdfor alli’sunderthesequencew=(! s ) 1 s=1 onwhichwearefocusingintheproof. Sincethesequence (! s ) 1 s=1 has been fixed now, the corresponding induced solutions sequence ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 isdeterministicastheresult. 49 Like what has been mentioned before, the N.E. of the game (2.15v) exists and is unique. Now, let’sprovethatthesequence ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 willconvergetothisuniqueN.E.. Letk·k represent the l 2 -norm. Under outer iteration v, for each inner iteration marked as µ with µ 0 and each player i, define ˜ H i µ,v (x i ,x i ) , 1 L v µ L v µ P s=1 H i (x i ,x i ,! s ). The inner iteration updatingschemeisgivenbelow: µ+1,v,i = argmax i 0 min x i 2 X i ˜ H i µ,v (x i ,x µ,v, i )+( i ) T 1 L v µ L v µ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 with x µ+1,v,i =argmin x i 2 X i ˜ H i µ,v (x i ,x µ,v, i )+( µ+1,v,i ) T 1 L v µ L v µ P s=1 G i (x i ,! s ) + c 2 ||x i ˜ x v 1,i || 2 c 2 || µ+1,v,i ˜ v 1,i || 2 . Usingthesameanalysisintheproofoflemma6,wehavethefollowinginequality: (x i x µ+1,v,i ) T (r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )+ k i P j=1 µ+1,v,i j 1 L v µ L v µ P s=1 r x iG i j (x µ+1,v,i ,! s )) +c·(x i x µ+1,v,i ) T (x µ+1,v,i ˜ x v 1,i ) ( i µ+1,v,i ) T 1 L v µ L v µ P s=1 G i (x µ+1,v,i ,! s ) +c·( i µ+1,v,i ) T ( µ+1,v,i ˜ v 1,i ) 0 8(x i , i )2X i ⇥ ⇤ i . Substituting (x i , i )by (x µ,v,i , µ,v,i ),wereachtheinequality(2.32)asbelow: (x µ,v,i x µ+1,v,i ) T (r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )+ 1 L v µ L v µ P s=1 k i P j=1 µ+1,v,i j r x iG i j (x µ+1,v,i ,! s ) +c·(x µ+1,v,i ˜ x v 1,i )) +( µ,v,i µ+1,v,i ) T ( 1 L v µ L v µ P s=1 G i j (x µ+1,v,i ,! s )+c·( µ+1,v,i ˜ v 1,i )) 0. (2.32) Applying the similar analysis on the inner iterationµ 1 whenµ 1 and substituting (x i , i ) by (x µ+1,v,i , µ+1,v,i ),wecanobtainanotherinequality: 50 (x µ+1,v,i x µ,v,i ) T (r x i ˜ H i µ 1,v (x µ,v,i ,x µ 1,v, i )+ 1 L v µ 1 L v µ 1 P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) +c·(x µ,v,i ˜ x v 1,i )) +( µ+1,v,i µ,v,i ) T ( 1 L v µ 1 L v µ 1 P s=1 G i j (x µ,v,i ,! s )+c·( µ,v,i ˜ v 1,i )) 0. (2.33) Combining(2.32)and(2.33)anddoingsomeorganizations,wehave: (x µ+1,v,i x µ,v,i ) T (r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )r x i ˜ H i µ 1,v (x µ,v,i ,x µ 1,v, i )) +c·kx µ+1,v,i x µ,v,i k 2 +c·k µ+1,v,i µ,v,i k 2  x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i T 0 B B B B B B B B B B B B @ 0 B B B B @ 1 L v µ 1 L v µ 1 P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 L v µ L v µ P s=1 k i P j=1 µ+1,v,i j r x iG i j (x µ+1,v,i ,! s ) 1 C C C C A 1 L v µ L v µ P s=1 G i (x µ+1,v,i ,! s ) 1 L v µ 1 L v µ 1 P s=1 G i (x µ,v,i ,! s ) 1 C C C C C C C C C C C C A . (2.34) The RHS of the inequality (2.34) is a bit hard to be analyzed directly because L v µ 1 is different fromL v µ and (x µ,v,i , µ,v,i )isdifferentfrom (x µ+1,v,i , µ+1,v,i )simultaneously. Tosimplifytheanalysis,weneedtoaddsomeintermediatevariableslikebelow. RHSof(2.34) = x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i T 0 B B B @ 1 L v µ 1 L v µ 1 P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 L v µ L v µ P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 L v µ L v µ P s=1 G i (x µ,v,i ,! s ) 1 L v µ 1 L v µ 1 P s=1 G i (x µ,v,i ,! s ) 1 C C C A 51 + x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i T 0 B B B B B B B @ 0 B B B @ 1 L v µ L v µ P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 L v µ L v µ P s=1 k i P j=1 µ+1,v,i j r x iG i j (x µ+1,v,i ,! s ) 1 C C C A 1 L v µ L v µ P s=1 G i (x µ+1,v,i ,! s ) 1 L v µ L v µ P s=1 G i (x µ,v,i ,! s ) 1 C C C C C C C A . (2.35) Recall that in each inner iterationµ, the random function: T µ,v,i (x i , i ), 1 L v µ L v µ P s=1 ( i ) T G i (x i ,! s ) isconvex-concaveinvariablesx i and i when (x i , i )2X i ⇥ ⇤ i . Also,wehave: thesecondpartoftheRHSof(2.35) =(x µ+1,v,i x µ,v,i ) T (r x iT µ,v,i (x µ,v,i , µ,v,i )r x iT µ,v,i (x µ+1,v,i , µ+1,v,i )) +( µ+1,v,i µ,v,i ) T (r iT µ,v,i (x µ+1,v,i , µ+1,v,i )r iT µ,v,i (x µ,v,i , µ,v,i )) = (x µ+1,v,i x µ,v,i ) T (r x iT µ,v,i (x µ+1,v,i , µ+1,v,i )r x iT µ,v,i (x µ,v,i , µ,v,i )) +( µ+1,v,i µ,v,i ) T (r iT µ,v,i (x µ+1,v,i , µ+1,v,i )r iT µ,v,i (x µ,v,i , µ,v,i )). (2.36) Thus, based on the monotonicity of the concatenation of corresponding positive and negative gra- dients ofT µ,v,i (x i , i ) inx i 2X i and i 2⇤ i respectively, we have that the RHS of (2.36) is less thanorequalto0. Hence,wehave: (x µ+1,v,i x µ,v,i ) T (r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )r x i ˜ H i µ 1,v (x µ,v,i ,x µ 1,v, i )) +c·kx µ+1,v,i x µ,v,i k 2 +c·k µ+1,v,i µ,v,i k 2  0 B B @ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i 1 C C A T 0 B B B B B B B B @ 0 B B B @ 1 L v µ 1 L v µ 1 P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 L v µ L v µ P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 C C C A 1 L v µ L v µ P s=1 G i (x µ,v,i ,! s ) 1 L v µ 1 L v µ 1 P s=1 G i (x µ,v,i ,! s ) 1 C C C C C C C C A 52  ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · 0 B B B B B B B B @ 0 B B B @ 1 L v µ 1 L v µ 1 P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 L v µ L v µ P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 C C C A 1 L v µ L v µ P s=1 G i (x µ,v,i ,! s ) 1 L v µ 1 L v µ 1 P s=1 G i (x µ,v,i ,! s ) 1 C C C C C C C C A  ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · 0 B B B @ 1 L v µ 1 L v µ 1 P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 L v µ L v µ P s=1 k i P j=1 µ,v,i j r x iG i j (x µ,v,i ,! s ) 1 C C C A + ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · 1 L v µ L v µ X s=1 G i (x µ,v,i ,! s ) 1 L v µ 1 L v µ 1 X s=1 G i (x µ,v,i ,! s ) = ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · ( µ,v,i ) T · 0 B B B @ 1 L v µ 1 L v µ 1 P s=1 r x iG i (x µ,v,i ,! s ) 1 L v µ L v µ P s=1 r x iG i (x µ,v,i ,! s ) 1 C C C A + ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · 1 L v µ L v µ X s=1 G i (x µ,v,i ,! s ) 1 L v µ 1 L v µ 1 X s=1 G i (x µ,v,i ,! s )  ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · µ,v,i · 0 B B @ 1 L v µ 1 L v µ 1 P s=1 r x iG i (x µ,v,i ,! s ) E[r x iG i (x µ,v,i ,!)] 1 C C A 53 + ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · µ,v,i · 0 B B @ E[r x iG i (x µ,v,i ,!)] 1 L v µ L v µ P s=1 r x iG i (x µ,v,i ,! s ) 1 C C A + ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · 1 L v µ L v µ X s=1 G i (x µ,v,i ,! s ) E[G i (x µ,v,i ,!)] + ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · E[G i (x µ,v,i ,!)] 1 L v µ 1 L v µ 1 X s=1 G i (x µ,v,i ,! s )  ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · µ,v,i · 0 B B @ sup x i 2 X i 0 B B @ 1 L v µ 1 L v µ 1 P s=1 r x iG i (x i ,! s ) E[r x iG i (x i ,!)] 1 C C A 1 C C A + ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · µ,v,i · 0 B B @ sup x i 2 X i 0 B B @ E[r x iG i (x i ,!)] 1 L v µ L v µ P s=1 r x iG i (x i ,! s ) 1 C C A 1 C C A + ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · 0 @ sup x i 2 X i 1 L v µ L v µ X s=1 G i (x i ,! s ) E[G i (x i ,!)] 1 A + ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ · 0 @ sup x i 2 X i E[G i (x i ,!)] 1 L v µ 1 L v µ 1 X s=1 G i (x i ,! s ) 1 A . (2.37) 54 The w we are considering here makes the equalities (2.19) to (2.22) hold. Also based on lemma 6, there exists a positive integer ¯ L w such that for any µ with L v µ ¯ L w , µ,v,i is bounded by a positive value independent of µ, v and i. Thus, from (2.37), we have that for any✏> 0, there exists a positive valueL 0 dependent on ✏ such that for anyµ withL v µ L 0 , we have the following inequalitieshold. (x µ+1,v,i x µ,v,i ) T (r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )r x i ˜ H i µ 1,v (x µ,v,i ,x µ 1,v, i )) +c·kx µ+1,v,i x µ,v,i k 2 +c·k µ+1,v,i µ,v,i k 2  ✏· x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i . ) (x µ+1,v,i x µ,v,i ) T (r x i ˜ H i µ,v (x µ+1,v,i ,x µ,v, i )r x i✓ i (x µ+1,v,i ,x µ,v, i )+r x i✓ i (x µ+1,v,i ,x µ,v, i ) (r x i ˜ H i µ 1,v (x µ,v,i ,x µ 1,v, i )r x i✓ i (x µ,v,i ,x µ 1,v, i ))r x i✓ i (x µ,v,i ,x µ 1,v, i )) +c·kx µ+1,v,i x µ,v,i k 2 +c·k µ+1,v,i µ,v,i k 2  ✏· x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i . Fromassumption(A1),equality(2.20)andthefactthatX i iscompact,whenµislargeenough,we 55 have: (x µ+1,v,i x µ,v,i ) T (r x i✓ i (x µ+1,v,i ,x µ,v, i )r x i✓ i (x µ,v,i ,x µ 1,v, i )) +c·kx µ+1,v,i x µ,v,i k 2 +c·k µ+1,v,i µ,v,i k 2  ✏· x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i +✏·kx µ+1,v,i x µ,v,i k 2✏· x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i . (2.38) By applying the mean value theory on the first term of the LHS in (2.38) and according to as- sumption(A2),itfollowsthat: ( ii +c)· x µ+1,v,i x µ,v,i 2 +c· µ+1,v,i µ,v,i 2  2✏· ✓ x µ+1,v,i x µ,v,i µ+1,v,i µ,v,i ◆ + X j6=i ij x µ,v,j x µ 1,v,j ! · x µ+1,v,i x µ,v,i (2.39) whereboththe ii and ij aredefinedintheassumption(A2)foralli=1,2,3,...,N andj6=i. From(2.39),wecanhavethefollowinginequality: ( ii +c) ⇣ ||x µ+1,v,i x µ,v,i || 2 + c ii +c || µ+1,v,i µ,v,i || 2 ⌘  2✏· q ii +c c · x µ+1,v,i x µ,v,i p c ii +c ( µ+1,v,i µ,v,i ) + P j6=i ij x µ,v,j x µ 1,v,j q c jj +c ( µ,v,j µ 1,v,j ) ! · x µ+1,v,i x µ,v,i p c ii +c ( µ+1,v,i µ,v,i ) . (2.40) 56 Suppose x µ+1,v,i x µ,v,i p c ii +c ( µ+1,v,i µ,v,i ) > 0. Thenbydividing ( ii +c)· x µ+1,v,i x µ,v,i p c ii +c ( µ+1,v,i µ,v,i ) on bothsidesoftheinequality(2.40),wehave: x µ+1,v,i x µ,v,i p c ii +c ( µ+1,v,i µ,v,i )  2✏· q 1 c·( ii +c) + P j6=i ij ii +c x µ,v,j x µ 1,v,j q c jj +c ( µ,v,j µ 1,v,j ) ! . (2.41) By noting that the inequality (2.41) holds when x µ+1,v,i x µ,v,i p c ii +c ( µ+1,v,i µ,v,i ) =0, we can always get (2.41)foreachplayeri. Whenweconcatenatetheinequalities(2.41)forallplayers,wehave: 0 B B B B B B B @ x µ+1,v,1 x µ,v,1 p c 11 +c ( µ+1,v,1 µ,v,1 ) . . . x µ+1,v,N x µ,v,N p c NN +c ( µ+1,v,N µ,v,N ) 1 C C C C C C C A  ˆ ⌥ 0 B B B B B B B @ x µ,v,1 x µ 1,v,1 p c 11 +c ( µ,v,1 µ 1,v,1 ) . . . x µ,v,N x µ 1,v,N p c NN +c ( µ,v,N µ 1,v,N ) 1 C C C C C C C A + 0 B B B B B B @ q 1 c·( 11 +c) ·2✏ . . . q 1 c·( NN +c) ·2✏ 1 C C C C C C A  ˆ ⌥ 0 B B B B B B B @ x µ,v,1 x µ 1,v,1 p c 11 +c ( µ,v,1 µ 1,v,1 ) . . . x µ,v,N x µ 1,v,N p c NN +c ( µ,v,N µ 1,v,N ) 1 C C C C C C C A + r c ll +c · 0 B B B B B B @ 2✏ . . . 2✏ 1 C C C C C C A (2.42) where ll , min i ii andthe ˆ ⌥ isdefinedas: ˆ ⌥ , 0 B B B B B B B B B B @ 0 12 c+ 11 ··· 1N c+ 11 11 c+ 22 0 ··· 1N c+ 22 . . . . . . . . . . . . 1N c+ NN ··· ··· 0 1 C C C C C C C C C C A . 57 From assumption (A2), we know that there exists a monotonic norm |·| d defined on theR N -space suchthattheinducednorm ˆ ⌥ d equals with< 1. Since in both the LHS and the RHS of (2.42), the vectors are composed by the non-negative componentvalues,bythemonotonicityofthenorm |·| d ,wehave: 0 B B B B B B B @ x µ+1,v,1 x µ,v,1 p c 11 +c ( µ+1,v,1 µ,v,1 ) . . . x µ+1,v,N x µ,v,N p c NN +c ( µ+1,v,N µ,v,N ) 1 C C C C C C C A d  ˆ ⌥ d · 0 B B B B B B B @ x µ,v,1 x µ 1,v,1 p c 11 +c ( µ,v,1 µ 1,v,1 ) . . . x µ,v,N x µ 1,v,N p c NN +c ( µ,v,N µ 1,v,N ) 1 C C C C C C C A d + r c ll +c · 0 B B B B B B @ 2✏ . . . 2✏ 1 C C C C C C A d = · 0 B B B B B B B @ x µ,v,1 x µ 1,v,1 p c 11 +c ( µ,v,1 µ 1,v,1 ) . . . x µ,v,N x µ 1,v,N p c NN +c ( µ,v,N µ 1,v,N ) 1 C C C C C C C A d + r c ll +c · 0 B B B B B B @ 2✏ . . . 2✏ 1 C C C C C C A d . (2.43) 58 Asthevalue✏isarbitrarilysmall,thevalue: r c ll +c · 0 B B B B B B @ 2✏ . . . 2✏ 1 C C C C C C A d canbearbitrarilysmallaswell. Let✏ 0 denotethisvalue. Now,fromtheaboveanalysesandtherelationshipspresentedin(2.43),wecanconcludethatthere existsapositiveintegerµ 0 suchthatforallµ µ 0 ,wehave: e µ+1,v  e µ,v +✏ 0 , where e µ+1,v , 0 B B B B B B B @ x µ+1,v,1 x µ,v,1 p c 11 +c ( µ+1,v,1 µ,v,1 ) . . . x µ+1,v,N x µ,v,N p c NN +c ( µ+1,v,N µ,v,N ) 1 C C C C C C C A d . Iteratingovertheaboveinequalitiesfromµtoµ 0 ,wededuce: e µ+1,v  µ µ 0 +1 e µ 0 ,v +✏ 0 µ µ 0 X k=0 k  µ µ 0 +1 e µ 0 ,v + ✏ 0 1 . Hence limsup µ!1 e µ,v  2✏ 0 1 . As ✏ 0 is arbitrarily small, we have lim µ!1 e µ,v =0. Thus, in any outer iterationv, we can establish the consecutive convergence property of the inner-loop solutions se- quence ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 inalmosteverywhere(a.e.). Since we have the boundedness ofx µ,v,i and µ,v,i over anyµ andi whenL v µ is large enough, the existence of the accumulation point of the inner-loop solutions sequence can be guaranteed. And fromtheaboveanalyses,weknowthatthegeneratedsolutionssequencewillhavetheconsecutive 59 convergence property. Hence, for any accumulation point, e.g. (¯ x v , ¯ v ), there exists one sub- sequence ((x µ k ,v,i , µ k ,v,i ) N i=1 ) 1 k=1 converging to (¯ x v , ¯ v ). Therefore, ((x µ k +1,v,i , µ k +1,v,i ) N i=1 ) 1 k=1 willconvergeto(¯ x v , ¯ v )aswell. Usingthesimilaranalysisintheproofoflemma6oneachinner iterationµ k ,wehave: (x i x µ k +1,v,i ) T (r x i ˜ H i µ k ,v (x µ k +1,v,i ,x µ k ,v, i )+ 1 L v µ k L v µ k P s=1 k i P j=1 µ k +1,v,i j r x iG i j (x µ k +1,v,i ,! s ) +c·(x µ k +1,v,i ˜ x v 1,i ))+( i µ k +1,v,i ) T ( 1 L v µ k L v µ k P s=1 G i j (x µ k +1,v,i ,! s ) +c·( µ k +1,v,i ˜ v 1,i )) 0 8x i 2X i , 8 i 0. Passing the limit ofk and noting thatw=(! s ) 1 s=1 makes the equalities (2.19) to (2.22) hold, for eachi,wehave: (x i ¯ x v,i ) T (r x i✓ i (¯ x v,i ,¯ x v, i )+ k i X j=1 ¯ v,i j ·E[r x iG i j (¯ x v,i ,!)]+c·(¯ x v,i ˜ x v 1,i )) +( i ¯ v,i ) T ( E[G i (¯ x v,i ,!)]+c·( ¯ v,i ˜ v 1,i )) 0 8x i 2X i , 8 i 0 which implies that (¯ x v , ¯ v ) solves the V.I. problem VI( ˜ K, ˜ F c,˜ z v 1) and hence by the equivalence between VI( ˜ K, ˜ F c,˜ z v 1) and the max-min game (2.15v), we conclude that (¯ x v , ¯ v ) is the unique N.E.ofthegame (2.15v)and(˜ x v , ˜ v )=(¯ x v , ¯ v ). Toextendtheconvergenceresultfromanarbitraryconvergentsubsequencetothewholesequence, wejustneedtonotethatthewholesolutionssequence ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 isbounded. Suppose ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 doesnotconvergetothisuniqueN.E.ofthegame(2.15v)inouteriterationv. Thenthereexistsapositivevalue✏suchthattheeventk(x µ,v,i , µ,v,i ) N i=1 (˜ x v,i , ˜ v,i ) N i=1 k>✏hap- pensininfinitelymanytimes. Therefore,wecanhaveasubsequence ((x µ k ,v,i , µ k ,v,i ) N i=1 ) 1 k=1 with 60 k(x µ k ,v,i , µ k ,v,i ) N i=1 (˜ x v,i , ˜ v,i ) N i=1 k>✏ for all µ k ’s. Since ((x µ k ,v,i , µ k ,v,i ) N i=1 ) 1 k=1 is bounded, theremustexistaconvergentsubsequenceof((x µ k ,v,i , µ k ,v,i ) N i=1 ) 1 k=1 ,e.g. ((x µ k l ,v,i , µ k l ,v,i ) N i=1 ) 1 l=1 . Basedontheconclusionwehaveestablishedabove, ((x µ k l ,v,i , µ k l ,v,i ) N i=1 ) 1 l=1 willalsoconvergeto (˜ x v,i , ˜ v,i ) N i=1 ,whichisacontradictiontothesettingthatk(x µ k l ,v,i , µ k l ,v,i ) N i=1 (˜ x v,i , ˜ v,i ) N i=1 k>✏ forallµ k l ’s. Hence,thewholesolutionssequenceconvergesto(˜ x v,i , ˜ v,i ) N i=1 . The sequence ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 is induced by (! s ) 1 s=1 which is an arbitrary random sequence making the equalities (2.19) to (2.22) hold. Hence, by assumption (A1.5), we conclude that the inner-loop solutions sequence ((x µ,v,i , µ,v,i ) N i=1 ) 1 µ=1 converges to the unique N.E. of the game (2.15v)ina.e.. ⇤ Now,wehaveestablishedtheconvergenceresultsforbothouterandinner-loopsintheLagrangian scheme. However,intherealimplementation,itisimpossibletoiterateeachloopininfinitelymany times. To make this algorithm more applicable, we need to find some stopping criteria for both outerandinner-loopssuchthatthedifferencebetweenthesolutionweobtainthroughthepractical implementationandtherealsolutioniswithinatolerablelevelinsomeprobabilisticsense. Oneavenuetoaccomplishthatistoinvestigatetheconvergenceratesofouterandinner-loopsand then combine them together to help us design a specific implementation scheme to obtain a solu- tioncloseenoughtotherealoneinsomeprobabilisticsense. Inthenextchapter,wewillgivetherateanalysisoftheLagrangianscheme. 61 3 ConvergenceRateAnalysisoftheLagrangianScheme In this chapter, we will investigate the convergence rates of outer and inner-loops and then use them to design the implementation in detail. In order to make the analysis easier, we simplify the model described in (2.1). The only change is to let each player’s objective function ✓ i (x i ,x i ) be deterministic. In another word, in this whole chapter, we assume ✓ i (x i ,x i )⌘ H i (x i ,x i ,!) for anyrealized!. Asaresult,thegameweareconsideringhereis: 8 > > < > > : min x i 2 X i ✓ i (x i ,x i ) s.t. E[G i (x i ,!)] 0 9 > > = > > ; N i=1 . (3.1) Correspondingly,theassumptions(A1)to(A3)describedinchapter2willbeupdatedasbelow. AssumptionsintheRateAnalysisoftheLagrangianScheme: (A’1): Intheoriginalgame(3.1),foreachplayeri,wehavetheassumptionsasfollows. (A’1.1): TheprivatesetX i isconvexandcompact. (A’1.2): Givenanyx i 2X i ,✓ i (x i ,x i )isstronglyconvexinx i 2X i . Also,✓ i (x i ,x i ) istwicedifferentiableinbothx i andx i overanopensetcontainingthedomain X = N Q i=1 X i . (A’1.3): Underanyrealized!,foranycomponentfunctionofG i (x i ,!),e.g. G i j (x i ,!) forj2{ 1,2,...,k i }, G i j (x i ,!)isconvexinx i andcontinuouslydifferentiable inx i overanopensetcontainingX i . (A’1.4): BothE[G i (x i ,!)]andE[r x iG i (x i ,!)]existandE[G i (x i ,!)]iscontinuously differentiableinx i withtheequalityr x iE[G i (x i ,!)] =E[r x iG i (x i ,!)]holding 62 foranyx i overanopensetcontainingX i . (A’1.5): BoththeG i (x i ,!)andthematrix-valuedrandomfunctionr x iG i (x i ,!)satisfy theuniformlawoflargenumbersoverX i . (A’1.6): Slater’sconditionholdsfortheconstraints: {x i 2X i | E[G i (x i ,!)] 0}, e.g.9 ˆ x i 2ri(X i )suchthatE[G i (ˆ x i ,!)]< 0. (A’2): DiagonalDominance(DD)Condition: Foreachplayeri,wedefine: (1) ii , inf x2 X smallesteigenvalueofr 2 x i ✓ i (x)2(0,1), (2) ij , sup x2 X r 2 x i x j ✓ i (x) 2 <1,forallj6=i; Thenthereexistsasetofpositiveconstants (d i ) N i=1 suchthat ii d i > P j6=i ij d j 8i=1,2,3,...,N. (A’3): ThefunctionF(Z)= 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 r x 1(✓ 1 (x 1 ,x 1 )+( 1 ) T E[G 1 (x 1 ,!)]) r 1(✓ 1 (x 1 ,x 1 )+( 1 ) T E[G 1 (x 1 ,!)]) . . . r x N(✓ N (x N ,x N )+( N ) T E[G N (x N ,!)]) r N(✓ N (x N ,x N )+( N ) T E[G N (x N ,!)]) 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 withZ = 2 6 6 6 6 6 6 6 6 6 6 4 x 1 1 x 2 2 . . . x N N 3 7 7 7 7 7 7 7 7 7 7 5 ismonotoneintheregionwherex i 2X i and i 0foralli’s. Sincethemodel(3.1)inthischapterisjustonespecialcaseofthemodel(2.1)inchapter2,allthe conclusions in chapter 2 remain valid in this chapter. Hence, all the convergence properties of the Lagrangianschemeprovedinthelastchapterstillholdhere. 63 Now,wecanstarttherateanalysisinthischapterformally. 3.1 OutlinesoftheRateAnalysis Priortotheformalanalysis,abigpictureofthelogicsshouldbeprovided. Remember that there are two loops in the Lagrangian scheme. In each outer iteration v where we parameterize ˜ z v 1 =(˜ x v 1,i , ˜ v 1,i ) N i=1 which is the solution in the (v 1)-th outer iteration, we solve the game (2.15v) by its inner-loop iterations. Then, in turn, we parameterize the v-th outer iteration’s solution ˜ z v =(˜ x v,i , ˜ v,i ) N i=1 in the (v+1)-th outer iteration and then start the corresponding inner-loop iterations to continue our algorithm iteratively. However, for each outer iterationv,wecannotimplementitsinneriterationsforinfinitelymanytimesandthusthesolution we can obtain at the stopping inner iteration may not be equal to the exact solution that can only beachievedbyimplementingtheinneriterationstothelimit. Inmathematics,ineachouteriterationv,weneedtosolvethe max-mingame (2.15v)below: ⇢ max i 0 min x i 2 X i ˜ L r i (x i ,x i , i ;˜ x v 1,i , ˜ v 1,i ) N i=1 . Ineachofitsinneriterationµ,foreachplayeri,wesolvethe max-minproblemshownbelow: max i 0 min x i 2 X i ✓ i (x i ,x µ,v, i )+( i ) T 1 L v µ L v µ X s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 andassignthesolutionpairto (x µ+1,v,i , µ+1,v,i ). We have shown that lim µ!1 (x µ,v,i , µ,v,i ) N i=1 =(˜ x v,i , ˜ v,i ) N i=1 which is the unique N.E. of the game (2.15v). 64 Supposeinthepracticalimplementation,westoptheinner-loopatiterationµandassign (x µ+1,v,i , µ+1,v,i ) N i=1 to (˜ x v,i , ˜ v,i ) N i=1 . Then the solution we have obtained, e.g (˜ x v,i , ˜ v,i ) N i=1 , is differentfromtherealN.E.ofthegame (2.15v),e.g. (¯ x v,i , ¯ v,i ) N i=1 . As the outer-loop is actually an implementation of the proximal point (P.P.) algorithm and the inner-loop is the combination of the best-response (BR) algorithm and the SAA method, to make thelatteranalysismoreclear,weneedtodefinethosetworelationsasbelow: (¯ x v , ¯ v ), (x pr (˜ x v 1 , ˜ v 1 ), pr (˜ x v 1 , ˜ v 1 )) where (x pr (˜ x v 1 , ˜ v 1 ), pr (˜ x v 1 , ˜ v 1 ))istheuniqueN.E.ofthegame (2.15v)parameterizedby (˜ x v 1 , ˜ v 1 ) and (˜ x v , ˜ v ), (x br (˜ x v 1 , ˜ v 1 ), br (˜ x v 1 , ˜ v 1 )), (x µ+1,v,i , µ+1,v,i ) N i=1 whereµisthestoppinginneriterationinv. In the Lagrangian scheme, starting from v=1 with (x 0 , 0 ) 2 ˜ K, theoretically, we can use the inner-looptoobtaintheuniqueN.E.ofthegame (2.15v)likebelow: (¯ x 1 , ¯ 1 )=(x pr (x 0 , 0 ), pr (x 0 , 0 )). However,inthepracticalimplementation,wecanjustobtain (x br (x 0 , 0 ), br (x 0 , 0 )) = (x µ+1,1,i , µ+1,1,i ) N i=1 with µ being the stopping inner iteration in the outer iterationv=1. Let(˜ x 1 , ˜ 1 ) denote this solution that we have obtained in the outer iteration v=1inthepracticalimplementation. Thus,wehave(˜ x 1 , ˜ 1 )=(x br (x 0 , 0 ), br (x 0 , 0 )). Asaresult,intheouteriterationv=2,inpracticalimplementation,wewillparameterize(˜ x 1 , ˜ 1 ), instead of (¯ x 1 , ¯ 1 ) in the game (2.15v) with v =2. And in this game, ideally, we hope we can obtain the theoretical N.E., e.g. (¯ x 2 , ¯ 2 )=(x pr (˜ x 1 , ˜ 1 ), pr (˜ x 1 , ˜ 1 )). However, due to the same reason, we can just obtain an approximation, e.g. (˜ x 2 , ˜ 2 )=(x br (˜ x 1 , ˜ 1 ), br (˜ x 1 , ˜ 1 )). And then 65 parameterizeitinthegame (2.15v)withv=3. Hence,inthepracticalimplementation,wewillhave(˜ x v , ˜ v )forv=1,2,...,iteratively. Let’scall (˜ x v , ˜ v )asthe“inexactsolution”whilecall(¯ x v , ¯ v )asthe“inexact-exactsolution”foreachv. Foreachouteriterationv,let’sdenotetheso-called“exactsolution”as(ˆ x v , ˆ v )whichistheexact theoretical solution in the outer iterationv. Hence, starting fromv=1 with(ˆ x 0 , ˆ 0 )=(x 0 , 0 )2 ˜ K, iteratively, for eachv, we have that(ˆ x v , ˆ v ) is the unique N.E. of the game (2.15v) parameter- ized by (ˆ x v 1 , ˆ v 1 ). In chapter 2, in the outer-loop convergence section, we have shown that the sequence(ˆ x v , ˆ v ) 1 v=1 willconvergetooneN.E.ofthe max-mingame(2.6). Inthischapter,wewanttoobtainanupperboundofthevalueofE[k(x ⇤ , ⇤ ) (˜ x v , ˜ v )k]foreach v where (x ⇤ , ⇤ ) is one N.E. of the game (2.6) which is the limit point of the exact outer-loop solutionssequence,e.g. (x ⇤ , ⇤ )= lim v!1 (ˆ x v , ˆ v ). Toobtainsuchanupperbound,wecansplitE[k(x ⇤ , ⇤ ) (˜ x v , ˜ v )k]usingthetriangleinequality likebelow. E[k(x ⇤ , ⇤ ) (˜ x v , ˜ v )k] E[k(x ⇤ , ⇤ ) (ˆ x v , ˆ v )k]+E[k(ˆ x v , ˆ v ) (¯ x v , ¯ v )k]+E[k(¯ x v , ¯ v ) (˜ x v , ˜ v )k]. (3.2) Based on the definitions of (ˆ x v , ˆ v ), (¯ x v , ¯ v ) and (˜ x v , ˜ v ) for eachv, we know that in the RHS of the inequality (3.2), the valuek(x ⇤ , ⇤ ) (ˆ x v , ˆ v )k is actually deterministic whose upper bound canbeobtainedviatheouter-looprateanalysis. Recallingthat (ˆ x v , ˆ v )=(x pr (ˆ x v 1 , ˆ v 1 ), pr (ˆ x v 1 , ˆ v 1 )) and (¯ x v , ¯ v )=(x pr (˜ x v 1 , ˜ v 1 ), pr (˜ x v 1 , ˜ v 1 )), theupperboundofthesecondtermintheRHSoftheinequality(3.2),e.g. E[k(ˆ x v , ˆ v ) (¯ x v , ¯ v )k], 66 canbeobtainedbyanalyzingthepropertyoftheproximalmappingP : ˜ K7! ˜ K withP(x, ), (x pr (x, ), pr (x, )). TheupperboundofthethirdtermintheRHSoftheinequality(3.2),e.g. E[k(¯ x v , ¯ v ) (˜ x v , ˜ v )k], canbeobtainedthroughtheinner-looprateanalysisatouteriterationv. Thefigure1belowsummarizesthebasiclogicsoftheanalysesusedinthischapter. Figure1: TheflowchartoftheLagrangianschemerateanalysis Infigure1,weusedifferentcolorstodifferentiate(ˆ x v , ˆ v ),(¯ x v , ¯ v )and(˜ x v , ˜ v )foreachv. Under eachv, to have the upper bound ofE[k(x ⇤ , ⇤ ) (˜ x v , ˜ v )k], we can analyze the upper bounds of k(x ⇤ , ⇤ ) (ˆ x v , ˆ v )kandE[k(¯ x v , ¯ v ) (˜ x v , ˜ v )k]throughtheouterandinner-loopsrateanalyses respectively. Then using the non-expansiveness property of the proximal mapping which will be proved in the latter section, we can obtain the upper bound of E[k(ˆ x v , ˆ v ) (¯ x v , ¯ v )k] via the upper bound of E[k(ˆ x v 1 , ˆ v 1 ) (˜ x v 1 , ˜ v 1 )k]. Since (ˆ x 0 , ˆ 0 )=(˜ x 0 , ˜ 0 )=(x 0 , 0 ), we can use the above splitting scheme to compute the upper bound of E[k(ˆ x v 1 , ˆ v 1 ) (˜ x v 1 , ˜ v 1 )k] iterativelyandthuscomputetheupperboundofE[k(ˆ x v , ˆ v ) (¯ x v , ¯ v )k]. Thenbycombiningwith 67 the outer and inner-loops rate analyses, we can have the upper bound ofE[k(x ⇤ , ⇤ ) (˜ x v , ˜ v )k] inouteriterationv eventually. Remark 4: We must note clearly that in our practical implementation, in eachv, neither (¯ x v , ¯ v ) nor(ˆ x v , ˆ v )canbeobtained. Instead,wecanonlyhavethesolution(˜ x v , ˜ v )ineachouteriteration. Both(¯ x v , ¯ v )and(ˆ x v , ˆ v )areconceptual. Thereasonwhyweintroduce(¯ x v , ¯ v )and(ˆ x v , ˆ v )here istousethemastheintermediatevariablestohavetheupperboundofE[k(x ⇤ , ⇤ ) (˜ x v , ˜ v )k]. Now,let’sintroducetheorganizationsoftherestcontentsinthischapterbelow. First, we give the outer-loop convergence rate analysis to have the upper bound ofk(x ⇤ , ⇤ ) (ˆ x v , ˆ v )kforeachv. Next,undereachv,wewillinvestigategivenapositivevalue✏,howtodesign ourinner-loopimplementationsuchthatE[k(¯ x v , ¯ v ) (˜ x v , ˜ v )k] ✏viatheinner-looprateanal- ysis. Thenweshowthenon-expansivenesspropertyoftheproximalmapping. Lastly,wecombine themtogethertoobtaintheupperboundofE[k(x ⇤ , ⇤ ) (˜ x v , ˜ v )k]foreachvanduseittodevelop amechanismtogeta✏-closesolutionintheexpectationsense. 3.2 Outer-LoopConvergenceRateAnalysis Intheouter-loop,weusethesolutionssequenceofthegames(2.15v)stoapproachoneN.E.ofthe max-min game (2.6). We have shown its convergence property in the convergence analysis in the lastchapter. Compared with the convergence analysis, the outer-loop’s rate analysis is a bit more complicated and some extra assumptions are needed. Actually, the convergence rate result mainly comes from 68 the chapter 12 in [12]. Here, we just give the outline of the proof after introducing some related conceptsandlistingtheextraassumptionsnecessaryintheproof. Recallthattheouter-looprateanalysisisdevelopedtofindanupperboundfork(ˆ x v , ˆ v ) (x ⇤ , ⇤ )k under each outer iteration v. And in each v, (ˆ x v , ˆ v ) is the unique N.E. of the max-min game (2.15v)parameterizedby(ˆ x v 1 , ˆ v 1 )orequivalently,(ˆ x v , ˆ v )istheuniquesolutionof VI( ˜ K, ˜ F c,ˆ z v 1) with ˆ z v 1 =(ˆ x v 1 , ˆ v 1 ). Also, (x ⇤ , ⇤ ) is one N.E. of the game (2.6) or equiva- lently,isonesolutionofVI( ˜ K, ˜ F). Hence,fromtheV.I.perspective,thesequence(ˆ x v , ˆ v ) 1 v=1 isa solutions sequence obtained by applying the proximal point (P.P.) algorithm. Actually, there have beensomerateresultsalreadyfortheP.P.algorithmin[12]. Tousetherelevantrateresultsdevelopedin[12],let’sintroducesomeconceptsbelow. Definition5: AmappingF : X7! YisLipschitzcontinuousifthereexistsapositivevalueL> 0 suchthatkF(x 1 ) F(x 2 )k L·kx 1 x 2 kforanyx 1 2Xandx 2 2X. Here,k·kisthel 2 -norm andLiscalledthe“LipschitzConstant”. Definition6: For each V.I. problem, e.g. VI(K, F), an associated natural mapF nat K : R n 7! R n is definedasF nat K (x),x Q K (x F(x))foranyx2R n where Q K (x F(x))istheprojectionof thepointx F(x)ontoK.kF nat K (x)kiscalledthe“naturalresidual”wherek·kisthel 2 -normon theR n -space. Definition 7: For each V.I. problem, e.g. VI(K, F), where the solutions set S , SOL(K,F) is non-empty, we say that S has a local error bound with the natural residualkF nat K (x)k, if9> 0 andr> 0,suchthatforanyx2R n withkF nat K (x)k ,wehavedist(x,S) r·kF nat K (x)kwhere 69 dist(x,S)isthedistancebetweenthepointxandthesetS. Again,k·kisthel 2 -normonR n . Definition8: A sequence (x v ) 1 v=1 is said to converge to ¯ x at least R-linearly, if9 a positive value aandapositivevalue✓< 1suchthatkx v ¯ xk a·✓ v foranyv. Using the definitions 5 7, we can list the extra assumptions needed in our outer-loop conver- gencerateanalysis. (H1): The V.I. problem VI( ˜ K, ˜ F) that is equivalent to the max-min game (2.6) has the mapping ˜ F thatisLipschitzcontinuouswiththeconstantLunderl 2 -normona ˜ K’ssubsetwherethewhole solutionssequenceoftheP.P.algorithmiscontained. (H2): Thenon-emptysetSOL( ˜ K, ˜ F)hasalocalerrorboundwiththenaturalresidual|| ˜ F nat ˜ K (x, )|| withtheconstants andr beingdefinedinthedefinition7. (H3): Underassumption(A’1.6),foreachplayeri,wecangetoneSlater’spointx 0i withthevalue ofE[G i (x 0i ,!)]beingcomputable. Remark5: Actually, based on [12], the sequence generated by applying the proximal point (P.P.) algorithmisbounded. Astheresult,intheouter-loopofouralgorithm,wejustneedtorestrictour attentiononaboundedsubsetof ˜ K. Consequently,theassumption(H1)willbesatisfiedautomat- ically aslong as ˜ F(x, ) is continuouslydifferentiable. As for(H2), by referringto [12], the local error bound assumption will be satisfied as long as some general conditions hold for both ˜ K and 70 ˜ F(x, ). Using those definitions and assumptions above and noting that the positive value c in our La- grangian scheme is actually the reciprocal of the constant c investigated at [12], we can formally givethetheoremoftheouter-loopconvergenceratebelow. Theorem3 Assume assumptions (A’1) to (A’3) and (H1) to (H3) hold. Let L be the Lipschitz constantofthemapping ˜ F(x, )andlet andr betheconstantsintheassumption(H2). Thenfor anyc> L, given an arbitrary initial point (x 0 , 0 )2 ˜ K triggering the outer-loop, the sequence (ˆ x v , ˆ v ) 1 v=1 will converge to one point (x ⇤ , ⇤ ) 2SOL( ˜ K, ˜ F) at least R-linearly. That is9 two computable constantsa andt witha> 0 and 1>t> 0 such thatk(ˆ x v , ˆ v ) (x ⇤ , ⇤ )k a·t v foranyouteriterationv 1. Proof: Under assumptions (A’1), (A’3), (H1) and (H3), the existences ofa andt can be proved by refer- ringtotheorem12.6.1andtheorem12.6.6in[12]. In theorem 12.6.1, we havea = q (z 0 ) ⌘ 1 1 1 q ⌘ 2 ⌘ 1 +⌘ 2 andt = q ⌘ 2 ⌘ 1 +⌘ 2 with ⌘ 1 and ⌘ 2 being two posi- tivevaluesandz 0 =(x 0 , 0 ), (z 0 ),kz 0 z ⇤ k 2 andz ⇤ =(x ⇤ , ⇤ )beingasolutionofVI( ˜ K, ˜ F) whichisthelimitoftheouter-loopsolutionssequence,e.g. (x ⇤ , ⇤ )= lim v!1 (ˆ x v , ˆ v ). From the analysis in P1180 of [12], we know in our case, ⌘ 1 = 1 2 and by referring to theorem 12.6.1, proposition 12.6.5 and exercise 4.8.4 in [12], we have that ⌘ 2 can be computed using the constants ,r,candL. Hencea p (z 0 )=kz 0 z ⇤ kand 1>t> 0. Therefore,theonlythingtobeshownisthecomputabilityof (z 0 ). Asitisimpossibletoknowwhatthepointz ⇤ =(x ⇤ , ⇤ )ispriortotheimplementation,wecannot 71 computeaexactly. However,wecanhaveitsupperboundbytheboundednessofSOL( ˜ K, ˜ F). RecallingthatthesetXiscompact,tohavetheboundednessofSOL( ˜ K, ˜ F),wejustneedtobound ⇤ forany (x ⇤ , ⇤ )2SOL( ˜ K, ˜ F). Fromtheassumption(A’1.6),weknowthattheSlater’sconditionholdsforeachplayeri’sexpected- valueconstraints. Letx 0i betheSlater’spointforeachplayeriwithE[G i (x 0i ,!)]beingcomputable mentionedinassumption(H3). In the last chapter, we have shown that (x ⇤ , ⇤ )2SOL( ˜ K, ˜ F) if and only if (x ⇤ , ⇤ ) is a N.E. of the max-min game (2.6). Now, suppose (x ⇤ , ⇤ ) 2SOL( ˜ K, ˜ F). Then for each player i, given x ⇤ , i 2X i , (x ⇤ ,i , ⇤ ,i )isasolutionofthefollowing max-minproblem: max i 0 min x i 2 X i ✓ i (x i ,x ⇤ , i )+( i ) T E[G i (x i ,!)]. Bylemma2,weknowthatx ⇤ ,i istheoptimalsolutionofthefollowingoptimizationproblem: min x i 2 X i ✓ i (x i ,x ⇤ , i ) s.t. E[G i (x i ,!)] 0. Letd ⇤ ,i betheoptimalvalueoftheabove max-minproblem. Thenbyreferringto[41],wehave: d ⇤ ,i = ✓ i (x ⇤ ,i ,x ⇤ , i )+( ⇤ ,i ) T E[G i (x ⇤ ,i ,!)] = min x i 2 X i ✓ i (x i ,x ⇤ , i )+( ⇤ ,i ) T E[G i (x i ,!)]  ✓ i (x 0i ,x ⇤ , i )+ k i X j=1 ⇤ ,i j E[G i j (x 0i ,!)] ✓ i (x 0i ,x ⇤ , i )+ max 1 j k i {E[G i j (x 0i ,!)]}·[ k i X j=1 ⇤ ,i j ] ) min 1 j k i { E[G i j (x 0i ,!)]}·[ k i X j=1 ⇤ ,i j ] ✓ i (x 0i ,x ⇤ , i ) d ⇤ ,i ) k i X j=1 ⇤ ,i j  ✓ i (x 0i ,x ⇤ , i ) d ⇤ ,i min 1 j k i { E[G i j (x 0i ,!)]} . TheSlater’sconditionimpliesthestrongduality,whichmeansd ⇤ ,i =p ⇤ ,i wherep ⇤ ,i = ✓ i (x ⇤ ,i ,x ⇤ , i ). 72 Letu i beacomputedupperboundof sup x2 X ✓ i (x i ,x i )andl i beacomputedlowerboundof inf x2 X ✓ i (x i ,x i ). Then by using the properties of X and ✓ i (x i ,x i ) implied by assumption (A’1) and the Weierstrass’ Theorem [4], both u i and l i exist and are finite. Hence, we have: k i P j=1 ⇤ ,i j  u i l i min 1 j k i { E[G i j (x 0i ,!)]} ,  i . For each i, by noting that the above u i and l i are independent of the selection of (x ⇤ , ⇤ ) 2 SOL( ˜ K, ˜ F), we have that for any point (x ⇤ , ⇤ )2SOL( ˜ K, ˜ F), (x ⇤ ,i , ⇤ ,i ) is contained in a com- pactconvexsetdenotedbyX i ⇥ K i withK i , { i 0 | i j   i forany j=1,2,...,k i }. Therefore, if we select the initial point (x 0 , 0 ) such that (x 0,i , 0,i )2X i ⇥ K i for eachi, we can easilycomputeanupperboundofthevaluek (z 0 )kandthushaveanupperboundofa,e.g. a 0 . Hence, we havek(ˆ x v , ˆ v ) (x ⇤ , ⇤ )k a 0 ·t v for any v, which concludes our proof by treating thevaluea 0 astheainthestatement. ⇤ Thetheorem3givesusacomputableupperboundofk(ˆ x v , ˆ v ) (x ⇤ , ⇤ )k. In the next section, we will have the inner-loop convergence rate analysis and see under an outer- iteration v, how to design our inner-loop implementation such that E[k(¯ x v , ¯ v ) (˜ x v , ˜ v )k]  ✏ where✏isafixedbutarbitrarilyselectedpositivevalue. 3.3 Inner-LoopConvergenceRateAnalysis Inthissection,wewillfirstfocusontherateanalysisoftheinner-loopandthenusetheinner-loop convergence rate results to design our inner-loop implementation. Throughout the whole section, weassumethatalltheconstantsdefinedinassumption(A’2)arecomputable. 73 3.3.1 Inner-LoopConvergenceRate Now,let’sinvestigatetheinner-loopconvergencerate. Inthepracticalimplementationintheouteriterationv,(˜ x v , ˜ v )equalsthesolutionobtainedinthe µ-th inner iteration, e.g. (x µ+1,v,i , µ+1,v,i ) N i=1 , where µ is the stopping inner iteration under the outer iterationv. Hence, to study the property ofE[k(¯ x v , ¯ v ) (˜ x v , ˜ v )k], we need to investigate thevaluesofE[k(¯ x v , ¯ v ) (x µ,v,i , µ,v,i )k]forallµ 0. Forthelateranalyses,weneedtogiveanewdefinitionoftheerrorweareconsideringhere. Definition9: Let ij bethevaluedefinedintheDDassumption(A’2)foreachi,j=1,2,3,...,N. And let k represent one player such that kk =max i=1,2,...,N ii . Also, let d denote the positive vector described in (A’2). Then under a fixed outer iteration v, given the value of the tuple (˜ x v 1 , ˜ v 1 )=(˜ x v 1,i , ˜ v 1,i ) N i=1 ,foreachinneriterationµ,definetheerror ˜ e v µ tobe: ˜ e v µ ,E 2 6 6 6 6 6 6 6 6 4 0 B B B B B B B @ q kk +c c (x µ,v,1 ¯ x v,1 ) q kk +c 11 +c ( µ,v,1 ¯ v,1 ) . . . q kk +c c (x µ,v,N ¯ x v,N ) r kk +c NN +c ( µ,v,N ¯ v,N ) 1 C C C C C C C A d (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 7 7 5 where(¯ x v , ¯ v )=(¯ x v,i , ¯ v,i ) N i=1 istheuniqueN.E.ofthegame (2.15v)parameterizedby (˜ x v 1 , ˜ v 1 ),orequivalently,(¯ x v , ¯ v )=(x pr (˜ x v 1 , ˜ v 1 ), pr (˜ x v 1 , ˜ v 1 )). Consequently,(¯ x v , ¯ v ) isdeterministicwhenthetuple(˜ x v 1,i , ˜ v 1,i ) N i=1 isgiven. Intheabovedefinition,wemustnotethattheexpectationusedhereistheconditionalexpectation. Since given (˜ x v 1,i , ˜ v 1,i ) N i=1 , the tuple (¯ x v,i , ¯ v,i ) N i=1 is deterministic, the expectation above is to 74 betakentothetuple (x µ,v,i , µ,v,i ) N i=1 only. Here,let’sgiveonebasiclemmasignificantinourlatteranalyses. Lemma7 ForanyrandomvariableX suchthatitsexpectationE[X]existsanditsvariance 2 , VAR(X) is finite, we have: E[| 1 n n P s=1 X s E[X]|] p n for any positive integern where is the standarddeviationofX and (X s ) n s=1 isthesequenceofthei.i.d. randomsamplesofX. Proof: Fixing a postive integer n, we have 0  VAR(| 1 n n P s=1 X s E[X]|) = E[| 1 n n P s=1 X s E[X]| 2 ] E 2 [| 1 n n P s=1 X s E[X]|]. SinceE[ 1 n n P s=1 X s E[X]] = 0,wecanhavethefollowingdeductions: E 2 [| 1 n n X s=1 X s E[X]|] E[| 1 n n X s=1 X s E(X)| 2 ] =E[( 1 n n X s=1 X s E[X]) 2 ] E 2 [ 1 n n X s=1 X s E[X]] =Var( 1 n n X s=1 X s E[X]) =Var( 1 n n X s=1 X s )= Var(X) n = 2 n . Eventually,wehaveE[| 1 n n P s=1 X s E[X]|] p n . ⇤ The conclusion in the above lemma is very general. The reason why we prove it again is to make ourproofmoreclearinthefollowingtheoremwhenthelemmaisused. Before we give the inner-loop rate results, the last thing remained is to introduce another assump- tionrelatedtothevariancesofthetherandomvariablesinourmodel. 75 (H4): ThereexistsonepositivevalueD suchthatforanyplayeri,wehave: sup x i 2 X i { sup j=1,2,3,...,k i VAR(G i j (x i ,!))} D and sup x i 2 X i { sup j=1,2,3,...,k i [sup t=1,2,3,...,n i VAR((r x i(G i j (x i ,!))) t )]} D. Now,wecangiveonetheoremregardingtotheinner-looprateresults. Theorem4 Assume assumptions (A’1) to (A’3) and (H4) hold. Then for any positive valuec and under any outer iterationv 1, lettingL v 0 denote the number of i.i.d. random samples generated in the 0-th inner iteration, we have that there exist computable positive values ˆ v ( ˜ v 1 ) > 0 and 0 < ˆ < 1 where ˆ v ( ˜ v 1 ) is dependent on ˜ v 1 and ˆ is independent of v such that ˜ e v µ  ˆ v( ˜ v 1 ) (1 ˆ ) p L v 0 + ˆ µ · ˜ e v 0 forallinneriterationsµ’s. Proof: From now, fix an outer iteration v. Then the tuple (˜ x v 1,i , ˜ v 1,i ) N i=1 is observed already. Let (¯ x v , ¯ v )=(¯ x v,i , ¯ v,i ) N i=1 betheuniqueN.E.ofthegame (2.15v)parameterizedby (˜ x v 1 , ˜ v 1 )=(˜ x v 1,i , ˜ v 1,i ) N i=1 . Thus, the tuple (¯ x v,i , ¯ v,i ) N i=1 is deterministic now. For each i, we have ¯ x v,i 2X i and ¯ v,i 0. From theorem 2, we know that(¯ x v,i , ¯ v,i ) is the limit point of the inner-loopsolutionssequencedenotedby (x µ,v,i , µ,v,i ) 1 µ=1 ina.e.. Now,let’srecaptheupdatingschemeineachinneriterationµ: µ+1,v,i = argmax i 0 min x i 2 X i ✓ i (x i ,x µ,v, i )+( i ) T 1 L v µ L v µ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 with x µ+1,v,i =argmin x i 2 X i ✓ i (x i ,x µ,v, i )+( µ+1,v,i ) T 1 L v µ L v µ P s=1 G i (x i ,! s ) + c 2 ||x i ˜ x v 1,i || 2 c 2 || µ+1,v,i ˜ v 1,i || 2 . We must note that in this proof, we will not make the sequence (! s ) 1 s=1 fixed to develop our 76 analysis. Instead,wewilltreat (! s ) L v µ s=1 toberandomforanyµ. Hence, (x µ,v,i , µ,v,i ) N i=1 israndom aswell. Using almost the same logics in the inner-loop convergence proof, for each playeri and any inner iterationµ,wehave: (¯ x v,i x µ+1,v,i ) T (r x i✓ i (x µ+1,v,i ,x µ,v, i )+ 1 L v µ L v µ X s=1 k i X j=1 µ+1,v,i j r x iG i j (x µ+1,v,i ,! s ) +c·(x µ+1,v,i ˜ x v 1,i )) +( ¯ v,i µ+1,v,i ) T ( 1 L v µ L v µ X s=1 G i j (x µ+1,v,i ,! s )+c·( µ+1,v,i ˜ v 1,i )) 0. (3.3) Notethatforeachi,given(¯ x v, i , ¯ v, i ),(¯ x v,i , ¯ v,i )isthesolutionofthe max-minproblembelow: max i 0 min x i 2 X i ✓ i (x i ,¯ x v, i )+( i ) T E[G i (x i ,!)]+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 . Hence, using the similar analysis that we developed in theorem 2 and the assumption (A’1.4), we have: (x µ+1,v,i ¯ x v,i ) T (r x i✓ i (¯ x v,i ,¯ x v, i )+ k i X j=1 ¯ v,i j ·E[r x iG i j (¯ x v,i ,!)]+c·(¯ x v,i ˜ x v 1,i )) +( µ+1,v,i ¯ v,i ) T ( E[G i (¯ x v,i ,!)]+c·( ¯ v,i ˜ v 1,i )) 0. (3.4) 77 Combining(3.3)and(3.4)anddoingsomeorganizations,wehave: ✓ x µ+1,v,i ¯ x v,i µ+1,v,i ¯ v,i ◆ T 0 B B @ r x i✓ i (x µ+1,v,i ,x µ,v, i )r x i✓ i (¯ x v,i ,¯ x v, i )+c·(x µ+1,v,i ¯ x v,i ) c·( µ+1,v,i ¯ v,i ) 1 C C A  ✓ x µ+1,v,i ¯ x v,i µ+1,v,i ¯ v,i ◆ T 0 B B B B B B B @ 0 B B B @ k i P j=1 ¯ v,i j ·E[r x iG i j (¯ x v,i ,!)] 1 L v µ L v µ P s=1 k i P j=1 µ+1,v,i j r x iG i j (x µ+1,v,i ,! s ) 1 C C C A 1 L v µ L v µ P s=1 G i (x µ+1,v,i ,! s ) E[G i (¯ x v,i ,!)] 1 C C C C C C C A . (3.5) Compared with the inequality (2.34) in the proof of theorem 2, in the above inequality, we just mainly substitute (x µ,v,i , µ,v,i ) by (¯ x v,i , ¯ v,i ) and r x i ˜ H i µ,v (x i ,x i ) (or r x i ˜ H i µ 1,v (x i ,x i )) by r x i✓ i (x i ,x i ) and then replace the sample average terms in iteration µ 1 by the expectation termsunderthevariable(¯ x v,i , ¯ v,i )ontheRHSrespectively. Hence,usingthesameanalysisinthe proofoftheorem2,throughtheinequality(3.5),wecanreachthefollowinginequality: ( ii +c)·kx µ+1,v,i ¯ x v,i k 2 +c· µ+1,v,i ¯ v,i 2  x µ+1,v,i ¯ x v,i µ+1,v,i ¯ v,i ·kT µ,v,i k+ P j6=i ij kx µ,v,j ¯ x v,j k ! ·kx µ+1,v,i ¯ x v,i k (3.6) where ij isdefinedintheassumption(A’2)foreachi,j=1,2,3,...,N and T µ,v,i , 0 B B B @ k i P j=1 ¯ v,i j ·E(r x iG i j (¯ x v,i ,!)) k i P j=1 ¯ v,i j · 1 L v µ L v µ P s=1 r x iG i j (¯ x v,i ,! s ) ! 1 L v µ L v µ P s=1 G i (¯ x v,i ,! s ) E(G i (¯ x v,i ,!)) 1 C C C A . Compared with the proof in theorem 2 where we substitutekT µ,v,i k by✏ using the uniform law of largenumbers,here,wekeepkT µ,v,i ktogetsomerateresultsinthelatteranalysis. 78 Thesubsequentanalysisissimilartotheproofintheorem(2). However,tomakethewholeanalysis moreunderstandable,wehighlightitagain. BydoingsomebasicmathematicaloperationsinboththeLHSandtheRHSintheinequality(3.6), wehave: ( ii +c) ⇣ ||x µ+1,v,i ¯ x v,i || 2 + c ii +c || µ+1,v,i ¯ v,i || 2 ⌘  q ii +c c · x µ+1,v,i ¯ x v,i p c ii +c ( µ+1,v,i ¯ v,i ) ·||T µ,v,i || + P j6=i ij x µ,v,j ¯ x v,j q c jj +c ( µ,v,j ¯ v,j ) ! · x µ+1,v,i ¯ x v,i p c ii +c ( µ+1,v,i ¯ v,i ) . Suppose x µ+1,v,i ¯ x v,i p c ii +c ( µ+1,v,i ¯ v,i ) > 0. Thenbydividing ( ii +c)· x µ+1,v,i ¯ x v,i p c ii +c ( µ+1,v,i ¯ v,i ) inboth sidesintheaboveinequality,wehave: ✓ x µ+1,v,i ¯ x v,i q c ii +c ( µ+1,v,i ¯ v,i ) ◆  s 1 c·( ii +c) · T µ,v,i + 0 @ X j6=i ij ii +c ✓ x µ,v,j ¯ x v,j q c jj +c ( µ,v,j ¯ v,j ) ◆ 1 A . When x µ+1,v,i ¯ x v,i p c ii +c ( µ+1,v,i ¯ v,i ) =0, the above inequality holds as well. Thus, it holds under all cases. 79 Thecombinedexpressionforallplayersi=1,2,···,N isshownbelow: 0 B B B B B B B @ x µ+1,v,1 ¯ x v,1 p c 11 +c ( µ+1,v,1 ¯ v,1 ) . . . x µ+1,v,N ¯ x v,N p c NN +c ( µ+1,v,N ¯ v,N ) 1 C C C C C C C A  ˆ ⌥ 0 B B B B B B B @ x µ,v,1 ¯ x v,1 p c 11 +c ( µ,v,1 ¯ v,1 ) . . . x µ,v,N ¯ x v,N p c NN +c ( µ,v,N ¯ v,N ) 1 C C C C C C C A + 0 B B B B B B @ q 1 c·( 11 +c) kT µ,v,1 k . . . q 1 c·( NN +c) T µ,v,N 1 C C C C C C A  ˆ ⌥ 0 B B B B B B B @ x µ,v,1 ¯ x v,1 p c 11 +c ( µ,v,1 ¯ v,1 ) . . . x µ,v,N ¯ x v,N p c NN +c ( µ,v,N ¯ v,N ) 1 C C C C C C C A + s 1 c·( ll +c) · 0 B B B B B B @ kT µ,v,1 k . . . T µ,v,N 1 C C C C C C A (3.7) where ll , min i ii andthe ˆ ⌥ isdefinedas: ˆ ⌥ , 0 B B B B B B B B B B @ 0 12 c+ 11 ··· 1N c+ 11 11 c+ 22 0 ··· 1N c+ 22 . . . . . . . . . . . . 1N c+ NN ··· ··· 0 1 C C C C C C C C C C A . For convenience in the latter analysis, here, we need to have more mathematical operations as the preparations. Define kk , max i ii . Then by multiplying q kk +c c on both the LHS and the RHS 80 of(3.7),wehave: 0 B B B B B B B @ q kk +c c (x µ+1,v,1 ¯ x v,1 ) q kk +c 11 +c ( µ+1,v,1 ¯ v,1 ) . . . q kk +c c (x µ+1,v,N ¯ x v,N ) r kk +c NN +c ( µ+1,v,N ¯ v,N ) 1 C C C C C C C A  ˆ ⌥ 0 B B B B B B B @ q kk +c c (x µ,v,1 ¯ x v,1 ) q kk +c 11 +c ( µ,v,1 ¯ v,1 ) . . . q kk +c c (x µ,v,N ¯ x v,N ) r kk +c NN +c ( µ,v,N ¯ v,N ) 1 C C C C C C C A + r kk +c c 2 ·( ll +c) · 0 B B B B B B @ kT µ,v,1 k . . . T µ,v,N 1 C C C C C C A . (3.8) AstheLHSof(3.8)isavectorcomposedbynon-negativecomponents,usingthemonotonicityof |·| d ,wehave: 0 B B B B B B B @ q kk +c c (x µ+1,v,1 ¯ x v,1 ) q kk +c 11 +c ( µ+1,v,1 ¯ v,1 ) . . . q kk +c c (x µ+1,v,N ¯ x v,N ) r kk +c NN +c ( µ+1,v,N ¯ v,N ) 1 C C C C C C C A d  ˆ ⌥ d · 0 B B B B B B B @ q kk +c c (x µ,v,1 ¯ x v,1 ) q kk +c 11 +c ( µ,v,1 ¯ v,1 ) . . . q kk +c c (x µ,v,N ¯ x v,N ) r kk +c NN +c ( µ,v,N ¯ v,N ) 1 C C C C C C C A d + r kk +c c 2 ·( ll +c) · 0 B B B B B B @ kT µ,v,1 k . . . T µ,v,N 1 C C C C C C A d . (3.9) The assumption (A’2) implies that | ˆ ⌥ | d is bounded above by a positive value strictly less than 1 81 that can be computed using all the constants defined in assumption (A’2). Let ˆ denote this upper bound. Hence,wehave | ˆ ⌥ | d < ˆ . Now,wecandeducethefollowinginequality: 0 B B B B B B B @ q kk +c c (x µ+1,v,1 ¯ x v,1 ) q kk +c 11 +c ( µ+1,v,1 ¯ v,1 ) . . . q kk +c c (x µ+1,v,N ¯ x v,N ) r kk +c NN +c ( µ+1,v,N ¯ v,N ) 1 C C C C C C C A d  ˆ · 0 B B B B B B B @ q kk +c c (x µ,v,1 ¯ x v,1 ) q kk +c 11 +c ( µ,v,1 ¯ v,1 ) . . . q kk +c c (x µ,v,N ¯ x v,N ) r kk +c NN +c ( µ,v,N ¯ v,N ) 1 C C C C C C C A d + r kk +c c 2 ·( ll +c) · 0 B B B B B B @ kT µ,v,1 k . . . T µ,v,N 1 C C C C C C A d . (3.9’) Now,atouteriterationv,giventhetuple(˜ x v 1,i , ˜ v 1,i ) N i=1 ,bothsidesof(3.9’)arerandomscalars whosevaluesdependon (! s ) L v µ s=1 generatedinthecurrentv. Asalltheaboveanalysesinthisproof are developed under the observation of (˜ x v 1,i , ˜ v 1,i ) N i=1 , by taking expectations in both sides of (3.9’)conditionedon(˜ x v 1,i , ˜ v 1,i ) N i=1 ,wehave: E 2 6 6 6 6 6 6 6 6 4 0 B B B B B B B @ q kk +c c (x µ+1,v,1 ¯ x v,1 ) q kk +c 11 +c ( µ+1,v,1 ¯ v,1 ) . . . q kk +c c (x µ+1,v,N ¯ x v,N ) r kk +c NN +c ( µ+1,v,N ¯ v,N ) 1 C C C C C C C A d (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 7 7 5 82  ˆ ·E 2 6 6 6 6 6 6 6 6 4 0 B B B B B B B @ q kk +c c (x µ,v,1 ¯ x v,1 ) q kk +c 11 +c ( µ,v,1 ¯ v,1 ) . . . q kk +c c (x µ,v,N ¯ x v,N ) r kk +c NN +c ( µ,v,N ¯ v,N ) 1 C C C C C C C A d (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 7 7 5 + q kk +c c 2 ·( ll +c) ·E 2 6 6 6 6 6 6 4 0 B B B B B B @ kT µ,v,1 k . . . T µ,v,N 1 C C C C C C A d (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 5 . (3.10) Now,itisthetimetocheckthepropertyof E 2 6 6 6 6 6 6 4 0 B B B B B B @ kT µ,v,1 k . . . T µ,v,N 1 C C C C C C A d (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 5 . Defined l , min 1 i N d i . Thenfromthedefinitionof |·| d ,wehave: 0 B B B B B B @ kT µ,v,1 k . . . T µ,v,N 1 C C C C C C A d =max 1 i N kT µ,v,i k d i  max 1 i N kT µ,v,i k d l  1 d l ( T µ,v,1 +···+ T µ,v,N ). TakingexpectationsinboththeLHSandtheRHSintheaboveinequalityconditionedon (˜ x v 1,i , ˜ v 1,i ) N i=1 ,weget: 83 E 2 6 6 6 6 6 6 4 0 B B B B B B @ kT µ,v,1 k . . . T µ,v,N 1 C C C C C C A d (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 5  1 d l (E[kT µ,v,1 k|(˜ x v 1,i , ˜ v 1,i ) N i=1 ]+··· +E[ T µ,v,N (˜ x v 1,i , ˜ v 1,i ) N i=1 ]). (3.11) Under any inner iteration µ, for each player i, based on the definition of T µ,v,i , we have the in- equalitybelow: kT µ,v,i k k i X j=1 0 @ ¯ v,i j ·kE[r x iG i j (¯ x v,i ,!)] 1 L v µ L v µ X s=1 r x iG i j (¯ x v,i ,! s )k 1 A +k 1 L v µ L v µ X s=1 G i (¯ x v,i ,! s ) E[G i (¯ x v,i ,!)]k. Implied by lemma 6, we know in each outer iteration v, given (˜ x v 1,i , ˜ v 1,i ) N i=1 , for all i’s, (| µ,v,i | l 1 ) 1 µ=1 willbeuniformlyboundedbyacomputableconstantreliedon( ˜ v 1,i ) N i=1 eventually in a.e.. Therefore, | ¯ v,i | l 1 will also be bounded by the same constant for all i’s. We let ¯ U v ( ˜ v 1 ) denote this constant where the representation used here is to highlight the dependence of ¯ U v on ˜ v 1 . Hence, ¯ v,i j  ¯ U v ( ˜ v 1 )foralli’sandj’s. Now,theinequalityabovecanbeextendedasbelow: kT µ,v,i k ¯ U v ( ˜ v 1 )· k i X j=1 0 @ kE[r x iG i j (¯ x v,i ,!)] 1 L v µ L v µ X s=1 r x iG i j (¯ x v,i ,! s )k 1 A +k 1 L v µ L v µ X s=1 G i (¯ x v,i ,! s ) E[G i (¯ x v,i ,!)]k. 84 Then taking expectations in both sides of the above inequality conditioned on (˜ x v 1,i , ˜ v 1,i ) N i=1 , wehave: E[kT µ,v,i k|(˜ x v 1,i , ˜ v 1,i ) N i=1 ]  ¯ U v ( ˜ v 1 )· k i X j=1 0 @ E[kE[r x iG i j (¯ x v,i ,!)] 1 L v µ L v µ X s=1 r x iG i j (¯ x v,i ,! s )k|(˜ x v 1,i , ˜ v 1,i ) N i=1 ] 1 A +E[k 1 L v µ L v µ X s=1 G i (¯ x v,i ,! s ) E[G i (¯ x v,i ,!)]k|(˜ x v 1,i , ˜ v 1,i ) N i=1 ]. (3.12) Now, take one player i as an example. For player i, both the 1 L v µ L v µ P s=1 G i (¯ x v,i ,! s ) E[G i (¯ x v,i ,!)] andE[r x iG i j (¯ x v,i ,!)] 1 L v µ L v µ P s=1 r x iG i j (¯ x v,i ,! s )arevectors. Thenbyreferringtolemma7andthe assumption (H4) and also using the fact that the value ofl 1 -norm is not smaller than the value of l 2 -norm for any vector in Euclidean space, we can deduce that there is a positive value ' v ( ˜ v 1 ) which is independent of µ and i but dependent on ¯ U v ( ˜ v 1 ), such that the RHS of (3.12) is less than or equal to 'v( ˜ v 1 ) p L v µ for any inner iterationµ. Then from the inequality (3.11), the following inequalitywillbeproducedautomatically: E 2 6 6 6 6 6 6 4 0 B B B B B B @ kT µ,v,1 k . . . T µ,v,N 1 C C C C C C A d (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 5  N·'v( ˜ v 1 ) d l · p L v µ 8µ. (3.13) Let ˜ e v µ betheerrordefinedinthedefinition6. Thenforanyinneriterationµ,usingtheinequalities (3.10)and(3.13),wehave: 85 ˜ e v µ+1  ˆ · ˜ e v µ + ˆ v( ˜ v 1 ) p L v µ where ˆ v ( ˜ v 1 ), q kk +c c 2 ·( ll +c) · N·'v( ˜ v 1 ) d l . Iterating the above inequality from the µ-th inner iteration to the 0-th inner iteration under outer iterationv,wehavethefollowingdeductions: ˜ e v µ+1  ˆ ˜ e v µ + ˆ v ( ˜ v 1 ) p L v µ =) ˜ e v µ+1  ˆ ˜ e v µ + ˆ v ( ˜ v 1 ) p L v µ  ˆ 2 ˜ e v µ 1 + ˆ ˆ v ( ˜ v 1 ) p L v µ 1 + ˆ v ( ˜ v 1 ) p L v µ ···  ˆ µ+1 ˜ e v 0 + ˆ v ( ˜ v 1 ) p L v µ + ˆ ˆ v ( ˜ v 1 ) p L v µ 1 +···+ ˆ µ ˆ v ( ˜ v 1 ) p L v 0  ˆ v ( ˜ v 1 ) p L v 0 + ˆ ˆ v ( ˜ v 1 ) p L v 0 +···+ ˆ µ ˆ v ( ˜ v 1 ) p L v 0 + ˆ µ+1 ˜ e v 0  ˆ v ( ˜ v 1 ) (1 ˆ ) p L v 0 + ˆ µ+1 ˜ e v 0 . Eventually,foranyµ 0,wehave: ˜ e v µ  ˆ v ( ˜ v 1 ) (1 ˆ ) p L v 0 + ˆ µ ˜ e v 0 . (3.14) ⇤ We call the property presented by the inequality (3.14) in the above proof as the “Partial Con- traction Property in the Sense of Conditional Expectation”. The word “Partial” highlights the existenceoftheterm ˆ v( ˜ v 1 ) (1 ˆ ) p L v 0 intheRHSof(3.14). Inthenextsub-section,wewilldesigntheinner-loopimplementationindetailtoselectthenumber 86 of generated i.i.d. random samples and the number of inner iterations in an outer iterationv such that given a positive value ✏, the unconditional expectationE[k(¯ x v , ¯ v ) (˜ x v,i , ˜ v,i )k] is less than orequalto✏. 3.3.2 Inner-LoopImplementationDesign Thetheorem4justgivesusthepartialcontractionpropertyinthesenseofconditionalexpectation. However, the purpose to develop the rate analyses in this chapter is to have a scheme by which given any positive value ✏, we can design the corresponding implementation in detail to obtain a ✏-close solution to one N.E. of the max-min game (2.6) in the expectation sense. To accomplish that, we must know in each outer iterationv, given any positive value ✏, how to design the imple- mentationinitsinner-looptohavea✏-closesolutiontotheuniqueN.E.ofthegame (2.15v)inthe unconditionalexpectationsense. One thing we must note here is that in each outer iteration v, given (˜ x v 1,i , ˜ v 1,i ) N i=1 , the value ˜ e v 0 is deterministic by recalling that (x 0,v,i , 0,v,i ) N i=1 =(˜ x v 1,i , ˜ v 1,i ) N i=1 . Hence, at the beginning of the outer iteration v, once the (˜ x v 1,i , ˜ v 1,i ) N i=1 has been observed, we can compute an upper boundof ˜ e v 0 usingtheconclusionoflemma6. To explain this point, in lemma 6, we obtain an upper bound of µ,v,i over alli’s whenµ is large enough. Supposeassumption(H3)holds. Thentheboundiscomputable. Hence,asthelimitpoint of (( µ,v,i ) N i=1 ) 1 µ=1 , the point ( ¯ v,i ) N i=1 must be bounded by the same upper bound as well. There- fore, we can use the bound obtained in lemma 6 and the boundedness property of X to have an upper bound of ˜ e v 0 . Since the bound in lemma 6 depends on the value of ˜ v 1 , ( ˜ v 1,i ) N i=1 , the ˜ e v 0 ’sboundwillalsodependon ˜ v 1 . Welet ˆ ⇧( ˜ v 1 )denotethisupperboundon ˜ e v 0 . Now,let’sgivetwomoredefinitionsoftheerrorsconsideredinthelatteranalysis. 87 Definition 10: Letd denote the positive vector described in (A’2). Then under a fixed outer iter- ation v, given the value of the tuple (˜ x v 1 , ˜ v 1 )=(˜ x v 1,i , ˜ v 1,i ) N i=1 , for each inner iteration µ, definetheerror ¯ e v µ as: ¯ e v µ ,E 2 6 6 6 6 6 6 4 0 B B B B B B @ x µ,v,1 ¯ x v,1 µ,v,1 ¯ v,1 . . . x µ,v,N ¯ x v,N µ,v,N ¯ v,N 1 C C C C C C A d (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 5 . Definition 11: Letd denote the positive vector described in (A’2). Then under a fixed outer iter- ation v, given the value of the tuple (˜ x v 1 , ˜ v 1 )=(˜ x v 1,i , ˜ v 1,i ) N i=1 , for each inner iteration µ, definetheerror ˆ e v µ as: ˆ e v µ ,E 2 6 6 6 6 6 6 4 0 B B B B B B @ x µ,v,1 ¯ x v,1 µ,v,1 ¯ v,1 . . . x µ,v,N ¯ x v,N µ,v,N ¯ v,N 1 C C C C C C A (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 5 =E 2 6 6 6 6 6 6 4 0 B B B B B B @ x µ,v,1 ¯ x v,1 µ,v,1 ¯ v,1 . . . x µ,v,N ¯ x v,N µ,v,N ¯ v,N 1 C C C C C C A (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 5 =E[ (x µ,v,i , µ,v,i ) N i=1 (¯ x v,i , ¯ v,i ) N i=1 (˜ x v 1,i , ˜ v 1,i ) N i=1 ]. Based on the definitions of ˜ e v µ and ¯ e v µ , using the monotonicity of |·| d , we have ¯ e v µ  ˜ e v µ for any 88 µandv. Letd m , max 1 i N d i . Thenwecanhavethefollowingdeductions: ˆ e v µ =E 2 6 6 6 6 6 6 4 0 B B B B B B @ x µ,v,1 ¯ x v,1 µ,v,1 ¯ v,1 . . . x µ,v,N ¯ x v,N µ,v,N ¯ v,N 1 C C C C C C A (˜ x v 1,i , ˜ v 1,i ) N i=1 3 7 7 7 7 7 7 5  E  ✓ x µ,v,1 ¯ x v,1 µ,v,1 ¯ v,1 ◆ +···+ ✓ x µ,v,N ¯ x v,N µ,v,N ¯ v,N ◆ (˜ x v 1,i , ˜ v 1,i ) N i=1 =d m ·E  1 d m ✓ ✓ x µ,v,1 ¯ x v,1 µ,v,1 ¯ v,1 ◆ +···+ ✓ x µ,v,N ¯ x v,N µ,v,N ¯ v,N ◆ ◆ (˜ x v 1,i , ˜ v 1,i ) N i=1  N ·d m ·E  1 d m ✓ max 1 i N ✓ ✓ x µ,v,i ¯ x v,i µ,v,i ¯ v,i ◆ ◆◆ (˜ x v 1,i , ˜ v 1,i ) N i=1  N ·d m ·E 2 4 0 @ max 1 i N 0 @ x µ,v,i ¯ x v,i µ,v,i ¯ v,i d i 1 A 1 A (˜ x v 1,i , ˜ v 1,i ) N i=1 3 5 =N ·d m · ¯ e v µ . Thus,wehave ˆ e v µ  N ·d m · ¯ e v µ  N ·d m · ˜ e v µ . Eventually,wehave: ˆ e v µ  N ·d m · ˜ e v µ  N ·d m · ˆ v ( ˜ v 1 ) (1 ˆ ) p L v 0 + ˆ µ ˜ e v 0 !  N ·d m · ˆ v ( ˜ v 1 ) (1 ˆ ) p L v 0 + ˆ µ ˆ ⇧( ˜ v 1 ) ! . (3.15) 89 Remark6: Intheassumption(A’2),wecanseethatthescaleofthevector (d i ) N i=1 canbearbitrar- ily large. In this assumption, only the ratios between differentd i ’s matter. Here, the value of ˆ e v µ is definitely independent of the specific scale of (d i ) N i=1 . Also, when we compute each of ˆ v ( ˜ v 1 ) and ˆ ⇧( ˜ v 1 ),thecorrespondingcomponentin(d i ) N i=1 isdividedbysomevaluethatisindependent of (d i ) N i=1 . Hence, both the LHS and the RHS in the inequality (3.15) are actually independent of thescaleof (d i ) N i=1 . Now we can introduce the inner-loop implementation scheme enabling us to achieve our goal in thissection. Scheme1: SchemeofInner-loopImplementation Suppose the assumptions (A’1) to (A’3) and (H3) to (H4) hold. In our implementation, in each outer iteration v, given the observation of (˜ x v 1,i , ˜ v 1,i ) N i=1 and a positive value ✏ as the in- puts, we first compute the values of ˆ v ( ˜ v 1 ) and ˆ ⇧( ˜ v 1 ) where ˆ v ( ˜ v 1 ) is defined in the proof of theorem 4. Then if N · d m · ˆ ⇧( ˜ v 1 )  ✏, let (˜ x v , ˜ v )=(˜ x v 1 , v 1 ). Otherwise, let L v 0 ( ˜ v 1 )= d ⇣ 2N·dm·ˆ v( ˜ v 1 ) (1 ˆ )✏ ⌘ 2 e and µ( ˜ v 1 )= dlog ˆ ⇣ ✏ 2N·dm· ˆ ⇧( ˜ v 1 ) ⌘ e. Next, generate the L v 0 ( ˜ v 1 )numberofi.i.d. randomsamples,e.g. (! s ) L v 0 ( ˜ v 1 ) s=1 ,andusethemtocarryoutthenumber ofµ( ˜ v 1 )inneriterations. Lastly,let(˜ x v,i , ˜ v,i ) N i=1 =(x µ( ˜ v 1 ),v,i , µ( ˜ v 1 ),v,i ) N i=1 . Remark 7: In scheme 1, it is possible to do nothing in the inner-loop, e.g. µ( ˜ v 1 )=0, and hence we can terminate the outer-loop at iteration v and take (˜ x v 1 , ˜ v 1 ) as the estimate of one N.E. of the game (2.6). To explain why that is reasonable, we note that this case happens if and onlyifN ·d m · ˆ ⇧( ˜ v 1 ) ✏. Byrecallingthat ˆ ⇧( ˜ v 1 )isanupperboundof ˜ e v 0 andtheinequality 90 ˆ e v 0  N ·d m · ˜ e v 0 holds, we have N ·d m · ˆ ⇧( ˜ v 1 ) ✏ implying ˆ e v 0  ✏, which means the point (˜ x v 1,i , ˜ v 1,i ) N i=1 isalreadyagoodapproximationofthepoint(¯ x v,i , ¯ v,i ) N i=1 . Weknow(¯ x v , ¯ v )is the unique N.E. of the game (2.15v) parameterized by(˜ x v 1 , ˜ v 1 ). Therefore, the small distance between those two points is a strong indication of the consecutive convergence in the proximal point (P.P.) algorithm with the initial point being (˜ x v 1 , ˜ v 1 ). Thus under this case, we can rea- sonably terminate our Lagrangian scheme and take (˜ x v 1 , ˜ v 1 ) as the eventual estimate of one N.E.ofthegame(2.6). Now,let’sdefinetheunconditionalexpectationerrorasfollows. Definition12: Underafixedouteriterationv,definetheerrore v asbelow: e v ,E 2 6 6 6 6 6 6 4 0 B B B B B B @ ˜ x v,1 ¯ x v,1 ˜ v,1 ¯ v,1 . . . ˜ x v,N ¯ x v,N ˜ v,N ¯ v,N 1 C C C C C C A 3 7 7 7 7 7 7 5 =E h (˜ x v,i , ˜ v,i ) N i=1 (¯ x v,i , ¯ v,i ) N i=1 i . Under the above scheme, using the theorem 4, we develop the following theorem to conclude the analysisinthissection. Theorem5 Assume assumptions (A’1) – (A’3), (H3) and (H4) hold. Then for any positive value c, under any outer iteration v 1, given a fixed but arbitrarily selected positive value ✏, if we implement the inner-loop following the scheme 1 described above with the ✏ and (˜ x v 1 , ˜ v 1 ) as theinputs,wehavee v  ✏. 91 Proof: Fix a positive value c. For any outer iteration v, both the values L v 0 ( ˜ v 1 ) and µ( ˜ v 1 ) are the functions of ˜ v 1 and thus the functions of (˜ x v 1 , ˜ v 1 ). Based on the details of scheme 1, when L v 0 = L v 0 ( ˜ v 1 ) and µ = µ( ˜ v 1 ), the RHS of the inequality (3.15) is less than or equal to ✏ re- gardless of the value of (˜ x v 1 , ˜ v 1 ). Thus, if µ=0, we can terminate our implementation and take(˜ x v 1 , ˜ v 1 ) as the estimated N.E. solution in the game (2.6). Based on the above definitions and the form of the inequality (3.15), we havee v  ✏ by taking the expectations on both sides of (3.15). While,whenµ> 0,bytakingexpectationsonbothsidesoftheinequality(3.15)withµ 1 beingthestoppinginneriteration,westillhavee v  ✏,whichconcludesourproof. ⇤ In the next section, we will investigate the non-expansiveness property of the proximal mapping P defined in this chapter. It turns out that the non-expansiveness of P is an important bridge to connectdifferentpartsinouranalysesinthischapter. 3.4 Non-ExpansivenessoftheProximalMapping Inthissection,wewillbuildtheboundofE[k(ˆ x v , ˆ v ) (¯ x v , ¯ v )k]foreachv. Recallingthat (ˆ x v , ˆ v )=(x pr (ˆ x v 1 , ˆ v 1 ), pr (ˆ x v 1 , ˆ v 1 )) =P(ˆ x v 1 , ˆ v 1 )and (¯ x v , ¯ v )=(x pr (˜ x v 1 , ˜ v 1 ), pr (˜ x v 1 , ˜ v 1 )) =P(˜ x v 1 , ˜ v 1 ) whereP istheproximalmappingdefinedinthischapter,wewillinvestigatethepropertyofP. Actually, the proximal mapping P turns out to be non-expansive provided that the assumptions (A’1)and(A’3)hold. Theproofisformallygiveninthefollowingtheorem. 92 Theorem6 Suppose assumptions (A’1) and (A’3) hold. Let (x, ) and (x 0 , 0 ) be two arbitrary pointsin ˜ K = N Q i=1 (X i ⇥ ⇤ i ). Thenwehave: kP(x, ) P(x 0 , 0 )k =k(x pr (x, ), pr (x, )) (x pr (x 0 , 0 ), pr (x 0 , 0 ))kk (x, ) (x 0 , 0 )k. Proof: Withoutlossanygenerality,wecanuse(˜ x v 1 , ˜ v 1 )and(ˆ x v 1 , ˆ v 1 )underafixedouteriteration v as the two points being considered in the analysis. Now, what we need to show isk(¯ x v , ¯ v ) (ˆ x v , ˆ v )kk (˜ x v 1 , ˜ v 1 ) (ˆ x v 1 , ˆ v 1 )k. BasedonthedefinitionofthemappingP,wehave: (¯ x v , ¯ v ), theuniqueN.E.ofthegame ⇢ argmax i 0 min x i 2 X i ✓ i (x i ,x i )+( i ) T E[G i (x i ,!)]+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 n i=1 and (ˆ x v , ˆ v ), theuniqueN.E.ofthegame ⇢ argmax i 0 min x i 2 X i ✓ i (x i ,x i )+( i ) T E[G i (x i ,!)]+ c 2 ||x i ˆ x v 1,i || 2 c 2 || i ˆ v 1,i || 2 n i=1 . Fromlemma5,wehavethat(¯ x v , ¯ v )solvesVI( ˜ K, ˜ F c,˜ z v 1)with ˜ z v 1 =(˜ x v 1 , ˜ v 1 )and(ˆ x v , ˆ v ) solvesVI( ˜ K, ˜ F c,ˆ z v 1)with ˆ z v 1 =(ˆ x v 1 , ˆ v 1 ). Also,wehave ˜ F c,˜ z v 1(Z)= ˜ F(Z)+c·(Z ˜ z v 1 )and ˜ F c,ˆ z v 1(Z)= ˜ F(Z)+c·(Z ˆ z v 1 )where ˜ F(Z)hasbeendefinedinchapter2as: 93 ˜ F(Z), 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 r x 1L 1 (x 1 ,x 1 , 1 ) r 1L 1 (x 1 ,x 1 , 1 ) . . . r x NL N (x N ,x N , N ) r NL N (x N ,x N , N ) 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 withZ = 2 6 6 6 6 6 6 6 6 6 6 4 x 1 1 x 2 2 . . . x N N 3 7 7 7 7 7 7 7 7 7 7 5 where L i (x i ,x i , i )= ✓ i (x i ,x i )+( i ) T E[G i (x i ,!)]foreachi. Assumption(A’3)impliesthemonotonicityof ˜ F(Z)inZ2 ˜ K. Let ¯ z v =(¯ x v , ¯ v )and ˆ z v =(ˆ x v , ˆ v ). Thenusingtheaboveanalysis,wehave: (¯ z v ˆ z v ) T ˜ F(ˆ z v )+c·(¯ z v ˆ z v ) T (ˆ z v ˆ z v 1 ) 0, (3.16) (ˆ z v ¯ z v ) T ˜ F(¯ z v )+c·(ˆ z v ¯ z v ) T (¯ z v ˜ z v 1 ) 0. (3.17) Combining(3.16)and(3.17)andusingthemonotonicityof ˜ F(Z)inZ2 ˜ K,wehave: c·(¯ z v ˆ z v ) T (ˆ z v ˆ z v 1 )+c·(ˆ z v ¯ z v ) T (¯ z v ˜ z v 1 ) (¯ z v ˆ z v ) T ( ˜ F(¯ z v ) ˜ F(ˆ z v )) 0. (3.18) By(3.18),wecanfurtherdeducethat: (¯ z v ˆ z v ) T (ˆ z v ˆ z v 1 )+(ˆ z v ¯ z v ) T (¯ z v ˜ z v 1 ) 0 ) (¯ z v ˆ z v ) T (˜ z v 1 ˆ z v 1 ) (¯ z v ˆ z v ) T (¯ z v ˆ z v )=k¯ z v ˆ z v k 2 )k ¯ z v ˆ z v k·k˜ z v 1 ˆ z v 1 kk ¯ z v ˆ z v k 2 )k ˜ z v 1 ˆ z v 1 kk ¯ z v ˆ z v k fromwhichwecanconcludeourproofbyrecallingthedefinitionsof ˜ z v 1 , ˆ z v 1 , ¯ z v and ˆ z v respec- tively. ⇤ Theabovetheoremsaysthatinourimplementation,foranyouteriterationv,wehave: 94 k(¯ x v , ¯ v ) (ˆ x v , ˆ v )kk (˜ x v 1 , ˜ v 1 ) (ˆ x v 1 , ˆ v 1 )k. Since (˜ x v 1 , ˜ v 1 ) is stochastic, (¯ x v , ¯ v ) is stochastic as well. Taking expectations on both sides oftheaboveinequality,weget: E[k(¯ x v , ¯ v ) (ˆ x v , ˆ v )k] E[k(˜ x v 1 , ˜ v 1 ) (ˆ x v 1 , ˆ v 1 )k]foranyv. Hence,tohaveanupperboundofE[k(¯ x v , ¯ v ) (ˆ x v , ˆ v )k]inthev-thouteriteration,wejustneed toobtaintheupperboundofE[k(˜ x v 1 , ˜ v 1 ) (ˆ x v 1 , ˆ v 1 )k]intheouteriterationv 1. By now, we have established all the necessary parts to accomplish the complexity analysis. In the next section, we will combine them together to develop a specific implementation scheme to achieve a ✏-close solution in the expectation sense with ✏ being a fixed but arbitrarily selected positivevalue. 3.5 TheSchemeinPracticalImplementation In the beginning of this chapter, we said that the main purpose of this chapter is to develop a scheme for the practical implementation such that given any positive value ✏, we can obtain a solution(˜ x v , ˜ v ) withE[k(˜ x v , ˜ v ) (x ⇤ , ⇤ )k] ✏ where (x ⇤ , ⇤ ) is a N.E. of the game (2.6). To do this, for eachv, we use the inequality (3.2) to get an upper bound ofE[k(˜ x v , ˜ v ) (x ⇤ , ⇤ )k], e.g. E[k(˜ x v , ˜ v ) (x ⇤ , ⇤ )k] E[k(x ⇤ , ⇤ ) (ˆ x v , ˆ v )k]+E[k(ˆ x v , ˆ v ) (¯ x v , ¯ v )k]+E[k(¯ x v , ¯ v ) (˜ x v , ˜ v )k]. Now,wehaveknownhowtoboundthevaluesofthethreetermsontheRHSintheaboveinequality respectively. To achieve our main goal, we need to combine the above analyses together to help us design 95 the implementation in detail. The following theorem gives us the outline of how to design our implementationtoachievethispurpose. Theorem7 Supposeassumptions(A’1)–(A’3)and(H1)–(H4)holdandalltheconstantsdefined inassumption(A’2)arecomputed. LetLbetheLipschitzconstantofthemapping ˜ F(Z)onZ2 ˜ K where ˜ F(Z)and ˜ K aredefinedintheequivalentV.I.problemofthegame(2.6). Selectcwithc>L andtheinitialpoint(x 0 , 0 )suchthat(x 0,i , 0,i )2X i ⇥ K i foreachiwithK i beingdefinedinthe proof of theorem 3. Let the three constantsa,t, and ˆ be defined the same as those in the theorem 3andthetheorem4respectivelyallof whichhavebeencomputedalready. Givenany positivevalue✏,ifa ✏,letD=0. Otherwise,letD =d log( ✏ 2a ) log(t) e. Maketheimplemen- tationsatisfythefollowingtwoconditions: i: Thetotal numberofouteriterationswerunequalsD. ii: IfD=0, take (x 0 , 0 ) as the estimate of one N.E. of the max-min game (2.6) and thus takex 0 as the estimate of the unique N.E. of the original game (2.1). Otherwise, starting fromv=1 and (˜ x 0 , ˜ 0 )=(x 0 , 0 ), in each outer iteration 1 v D, using ✏ 2D as the tolerable error controlled in the inner-loop of outer iterationv and (˜ x v 1 , ˜ v 1 ) as the parameter, we apply the scheme 1 to obtain(˜ x v , ˜ v ) andthenupdatev v+1 tocontinuetheimplementationuntilv =D+1. ThenwehaveE[k(˜ x D , ˜ D ) (x ⇤ , ⇤ )k] ✏ andthusE[k˜ x D x ⇤ k] ✏. Proof: First, let’s check the case when D =0. Under this case, we have a  ✏. In the proof of theo- rem 3, we have shownk(x 0 , 0 ) (x ⇤ , ⇤ )k a. Hence, when D=0,k(˜ x 0 , ˜ 0 ) (x ⇤ , ⇤ )k = k(x 0 , 0 ) (x ⇤ , ⇤ )k ✏,whichmeansE[k(˜ x D , ˜ D ) (x ⇤ , ⇤ )k] ✏. WhenD> 0,whichisequivalenttothata>✏,usingthevalueofD,e.g. D =d log( ✏ 2a ) log(t) e,wehave 96 k(ˆ x v , ˆ v ) (x ⇤ , ⇤ )k a·t D  ✏ 2 . In each outer iterationv with 1 v D, since we use ✏ 2D as the tolerable error controlled in the inner-loop and (˜ x v 1 , ˜ v 1 ) as the parameter to apply the scheme 1 to obtain (˜ x v , ˜ v ), we must haveE[k(˜ x v , ˜ v ) (¯ x v , ¯ v )k] ✏ 2D . Then starting fromv=1, we haveE[k(˜ x 1 , ˜ 1 ) (ˆ x 1 , ˆ 1 )k]= E[k(˜ x 1 , ˜ 1 ) (¯ x 1 , ¯ 1 )k] ✏ 2D by recallingthat(ˆ x 1 , ˆ 1 )=(¯ x 1 , ¯ 1 ),whichisbecause(ˆ x 0 , ˆ 0 )=(¯ x 0 , ¯ 0 )=(x 0 , 0 ). Andforv=2,wehave: E[k(˜ x 2 , ˜ 2 ) (ˆ x 2 , ˆ 2 )k] E[k(˜ x 2 , ˜ 2 ) (¯ x 2 , ¯ 2 )k]+E[k(¯ x 2 , ¯ 2 ) (ˆ x 2 , ˆ 2 )k]. Usingthenon-expansivenesspropertyoftheproximalmapping,wehave: E[k(˜ x 2 , ˜ 2 ) (ˆ x 2 , ˆ 2 )k] E[k(˜ x 2 , ˜ 2 ) (¯ x 2 , ¯ 2 )k]+E[k(˜ x 1 , ˜ 1 ) (ˆ x 1 , ˆ 1 )k] ✏ 2D + ✏ 2D . Whenv=3,similarlywehave: E[k(˜ x 3 , ˜ 3 ) (ˆ x 3 , ˆ 3 )k] E[k(˜ x 3 , ˜ 3 ) (¯ x 3 , ¯ 3 )k]+E[k(¯ x 3 , ¯ 3 ) (ˆ x 3 , ˆ 3 )k] E[k(˜ x 3 , ˜ 3 ) (¯ x 3 , ¯ 3 )k]+E[k(˜ x 2 , ˜ 2 ) (ˆ x 2 , ˆ 2 )k] ✏ 2D + ✏ 2D + ✏ 2D . Theniteratively,weget: E[k(˜ x D , ˜ D ) (ˆ x D , ˆ D )k] D · ✏ 2D = ✏ 2 . Thus,E[k(˜ x D , ˜ D ) (x ⇤ , ⇤ )k] E[k(˜ x D , ˜ D ) (ˆ x v , ˆ v )k]+k(ˆ x v , ˆ v ) (x ⇤ , ⇤ )k ✏ 2 + ✏ 2 = ✏. ⇤ As the conclusion of this chapter, let’s give the summarized description for the implementation indetail. Suppose the assumptions mentioned in the statement of theorem 7 are all satisfied and all of the corresponding constants have been computed before the start of the implementation. Then given an arbitrary positive value ✏, to guarantee E[k(˜ x v , ˜ v ) (x ⇤ , ⇤ )k]  ✏ where v is the last outer 97 iteration,wedeveloptheimplementationfollowingtheschemedescribedasbelow. ImplementationDetailsfortheLagrangianScheme Fix a positive value c withc> L where L is the Lipschitz constant of the mapping ˜ F(Z) on Z 2 ˜ K. Computed m =max 1 i N d i with (d i ) N i=1 being defined in assumption (A’2). Select an initial point (˜ x 0 , ˜ 0 )=(x 0 , 0 ) such that (˜ x 0,i , ˜ 0,i ) 2X i ⇥ K i for any i. Given a positive value ✏, if a ✏, take (˜ x 0 , ˜ 0 ) as the estimate of one N.E. of the game (2.6). Otherwise, let D =d log( ✏ 2a ) log(t) e. Startingfromv=1,wehave: (1)Setµ=0andlet (x 0,v,i , 0,v,i ) N i=1 =(˜ x v 1 , ˜ v 1 ). Also,usethevalue(˜ x v 1 , ˜ v 1 )tocompute the corresponding values ˆ v ( ˜ v 1 ) and ˆ ⇧( ˜ v 1 ) defined in the inner-loop convergence rate analy- sis. Let✏ 0 = ✏ 2D . IfN ·d m · ˆ ⇧( ˜ v 1 ) ✏ 0 ,weterminatetheimplementationandtake(˜ x v 1 , ˜ v 1 ) as the approximation of one N.E. of the game (2.6). Otherwise, letL v 0 =d ⇣ 2N·dm·ˆ v( ˜ v 1 ) (1 ˆ )✏ 0 ⌘ 2 e and ¯ µ v =dlog ˆ ⇣ ✏ 0 2N·dm· ˆ ⇧( ˜ v 1 ) ⌘ e. ThenwegenerateL v 0 i.i.d. randomsamplesof!,e.g. (! s ) L v 0 s=1 . (2)Whenµ ¯ µ v , weproceed tostep(4). Otherwise, fori=1,...,N, define (x µ+1,v,i , µ+1,v,i ) as follows: µ+1,v,i , argmax i 0 min x i 2 X i ✓ i (x i ,x µ,v, i )+( i ) T 1 L v 0 L v 0 P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 with x µ+1,v,i , arg min x i 2 X i ✓ i (x i ,x µ,v, i )+( µ+1,v,i ) T 1 L v 0 L v 0 P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || µ+1,v,i ˜ v 1,i || 2 . (3)Updateµ µ+1andreturnto(2). (4)Fori=1,...,N,assign ˜ x v,i =x µ,v,i and ˜ v,i = µ,v,i . (5)Ifv D,weterminatethisimplementationandtake(˜ x v , ˜ v )=(˜ x v,i , ˜ v,i ) N i=1 astheestimated 98 N.E.solutionofthegame(2.6). Otherwise,updatev v+1andreturnto(1). Thestepsfrom(1)to(4)intheabovedescriptionareactuallythestepsdescribedinscheme1. Here, wejustre-writethemtomakethedescriptionmoreclear. Remark 8: In the proof of theorem 3, we have shown that under assumptions (A’1) and (H3), the set SOL( ˜ K, ˜ F) is bounded by a computable compact set in the sense that for any (x ⇤ , ⇤ ) 2 SOL( ˜ K, ˜ F), we have (x ⇤ ,i , ⇤ ,i ) 2X i ⇥ K i for any i. As a result, the max-min game (2.6) is equivalenttothefollowingmodified max-mingame: ⇢ max i 2 K i min x i 2 X i L i (x i ,x i , i ), ✓ i (x i ,x i )+( i ) T E[G i (x i ,!)] N i=1 . (2.6’) Sinceintheproofsfromlemma2tolemma5,wedonotusetheunboundednessof⇤ i foranyi,all theconclusionsinthoselemmasstillholdifwesubstitute⇤ i byK i foreachi. Oneconsequenceof thismodificationisthatthelemma6isnolongerneededduetotheboundednessofK i . However, the almost everywhere convergence property of the inner-loop solutions sequence proved in the theorem2remainsvalid. Inthischapter,theboundednessofK i foranyimakesboth ˆ v ( ˜ v 1 )and ˆ ⇧( ˜ v 1 )beindependent of ˜ v 1 andthusenablesustocomputeL v 0 ( ˜ v 1 )andµ( ˜ v 1 )before ˜ v 1 isobserved. The reason why we do not use this modified version in this thesis is that when we conducted the simulation in the Lagrangian scheme under this modification, we found the modification did not bring any benefit in both the time efficiency and the solution’s accuracy. Also, adding the bound onthe partwouldmaketheoptimizationproblemsmorecomplicatedinthesimulation. 99 4 ⇢ MethodandItsConvergenceProperty Inthepreviouschapters,weintroducedtheLagrangianschemetosolvethestochasticgameswith expected-valueconstraints. Also,weshoweditsconvergencepropertyandgavethecorresponding convergencerateanalysis. In this chapter, we will introduce another method called the “⇢ Method". It turns out that the ⇢ method can achieve a better simulation performance compared to the Lagrangian scheme. How- ever, it is very hard to have the rate analysis. Thus, we present the two methods in this thesis. 4.1 FormulationandAssumptions Asthefirstpartinthischapter,let’sgivetheformulationandassumptionsforthe⇢ method. Com- paredwiththemodelstudiedintheLagrangianscheme,themodelconsideredhereisverysimilar. However, there are some assumptions different from what we have made in chapters 2 and 3. Hence, we will give the complete model formulation and assumptions for the ⇢ method in this section. 4.1.1 Formulation In this model, we have N players and each player i has a private strategy set X i ✓ R n i . Define X , N Q i=1 X i and X i , Q j6=i X j . The sub-problem of each player contains some expected-value functions. To include the random elements in our model, we introduce a d-dimensiontal random vector ! defined on a probability space (⌦ ,F,IP). The explanations of⌦ ,F and IP have been given in chapter 2. For each player i, his/her strategy is represented as a n i -dimensional vector x i 2R n i and he/she has an expected-valued objective function E[H i (x i ,x i ,!)] which depends 100 onallplayers’strategies(x i ) N i=1 . HereH i :R n ⇥ R d 7! Risarandomfunctionwithn, N P i=1 n i and we assume thatE[H i (x i ,x i ,!)] exists for anyx2X. Also the constraints on playeri’s strategy are private and described as {x i 2X i |E[G i (x i ,!)]  0} with G i (x i ,!) being a k i -dimensional vector-valuedrandomfunctionthathasalreadybeendefinedinchapter2. Thespecificdescriptions andassumptionsofthosemodelelementswillbeprovidedinthefollowingcontents. Foreachplayeri,define✓ i (x i ,x i ),E[H i (x i ,x i ,!)]. Thentheformulationofthegamecanbe summarizedasfollows: 8 > > < > > : min x i 2 X i ✓ i (x i ,x i ),E[H i (x i ,x i ,!)] s.t. E[G i (x i ,!)] 0 9 > > = > > ; N i=1 . (4.1) ANashEquilibrium(N.E.)isatuplex ⇤ , (x ⇤ ,i ) N i=1 2X, N Q i=1 X i suchthat x ⇤ ,i 2arg min x i 2 Z i ✓ i (x i ,x ⇤ , i )whereZ i , {x i 2X i |E[G i (x i ,!)] 0}foralli=1,···,N. By now, we have completely formulated the model considered in the ⇢ method. Before we talk aboutthedetails,let’sgivetheassumptionsofthismodelsufficientforthe⇢ methodtowork. 4.1.2 Assumptions (B1): Inthegame(4.1),foreachplayeri,wehavetheassumptionsasfollows. (B1.1): TheprivatesetX i isconvexandcompact. (B1.2): Underanyrealizedrandomsample!,H i (x i ,x i ,!)isstronglyconvexinx i 2X i givenanyx i 2X i = Q j6=i X j andtwicedifferentiableinxoveranopenset containingthedomainX = N Q i=1 X i . (B1.3): Underanyrealizedrandomsample!,foranycomponentfunctionofG i (x i ,!), 101 e.g. G i j (x i ,!)forj=1,2,...,k i , G i j (x i ,!)isconvexinx i 2X i . (B1.4): Foranyx i 2X i ,E[G i (x i ,!)]exists. Alsoforanyx2X,alltheE[H i (x i ,x i ,!)], E[r x H i (x i ,x i ,!)]andE[r 2 x H i (x i ,x i ,!)]existand ✓ i (x i ,x i ),E[H i (x i ,x i ,!)]istwicedifferentiableinxwiththeequalities r x ✓ i (x i ,x i ),r x E[H i (x i ,x i ,!)] =E[r x H i (x i ,x i ,!)] andr 2 x ✓ i (x i ,x i ),r 2 x E[H i (x i ,x i ,!)] =E[r 2 x H i (x i ,x i ,!)] holdingforanyxoveranopensetcontainingX. (B1.5): Boththevector-valuedrandomfunctionG i (x i ,!)andthescalar-valuedrandom functionH i (x i ,x i ,!)satisfytheuniformlawoflargenumbersoverX i andX respectively. (B1.6): Slater’sconditionholdsfortheconstraints: {x i 2X i | E[G i (x i ,!)] 0}, e.g.9 ˆ x i 2ri(X i )suchthatE[G i (ˆ x i ,!)]< 0. (B2): DiagonalDominance(DD)Condition: Foranyrealizedrandomsample! andeachplayeri,wedefine: (1) ii (!), inf x2 X smallesteigenvalueofr 2 x i H i (x i ,x i ,!)2(0,1), (2) ij (!), sup x2 X r 2 x i x j H i (x i ,x i ,!) 2 <1,forallj6=i; Thenthereexistsasetofpositiveconstants (d i ) N i=1 suchthatforanyrealized!, wehave: ii (!)d i > P j6=i ij (!)d j 8i=1,2,3,...,N. Remark 9: The assumptions here have 3 main differences compared to the assumptions in the Lagrangian scheme. The first one is in (B1), for each player i, given any realized random vector !, it does not require the differentiability in x i for each component of the vector-valued random 102 function G i (x i ,!), e.g. G i j (x i ,!) for j =1,2,...,k i . The second one is (A2) only assumes the DDconditioninexpectationform,while(B2)requiresittoholdoverall!’s. Thethirdoneisthat thereisnoassumptionanalogousto(A3)inthe⇢ method. The assumption (B2) is very strong. Actually, we can prove that it implies the assumption (A2) in chapter 2 provided that an extended version of assumption (B1) holds. We provide the formal proofofthispointinthefollowinglemma. Lemma8 Suppose the assumption (B1) holds and also for each i,r 2 x H i (x i ,x i ,!) satisfies the uniformlawoflargenumbersoverX. Thentheassumption(B2)implies(A2). Proof: Let’s fix a playeri and a pointx2X. Also, we focus on a random sequence, e.g. (! s ) 1 s=1 , such thatthefollowingequalityholdsunderthesequence (! s ) 1 s=1 : lim L!1 [sup x2 X || 1 L L P s=1 r 2 x H i (x i ,x i ,! s ) E[r 2 x H i (x i ,x i ,!)]|| 2 ]=0. (4.2) Since we have assumed thatr 2 x H i (x i ,x i ,!) satisfies the uniform law of large numbers over X, theexistenceofsuchrandomsequence (! s ) 1 s=1 canbeguaranteed. For any s=1,2,...,1, the matrixr 2 x i H i (x i ,x i ,! s ) is symmetric and by assumption (B2), we knowthatthesmallesteigenvalueofitispositive,whichmeansitispositive-definite. Hence,with anypositiveintegerL,thematrix 1 L L P s=1 r 2 x i H i (x i ,x i ,! s )ispositivedefiniteaswellandtherefore 0< min ( 1 L L P s=1 r 2 x i H i (x i ,x i ,! s )) = 1 L min ( L P s=1 r 2 x i H i (x i ,x i ,! s )) with min (M) representing thesmallesteigenvalueofMwhereMisanarbitrarysymmetricandpositivedefinitematrix. UsingtheWeyl’sinequality[15],wehave: min ( 1 L L P s=1 r 2 x i H i (x i ,x i ,! s )) = 1 L min ( L P s=1 r 2 x i H i (x i ,x i ,! s )) 1 L L P s=1 min (r 2 x i H i (x i ,x i ,! s )). 103 Thus,wehave: min ( 1 L L P s=1 r 2 x i H i (x i ,x i ,! s )) 1 L L P s=1 inf x2 X min (r 2 x i H i (x i ,x i ,! s )) = 1 L L P s=1 ii (! s ). Basedonassumption(B2),wecandeducethefollowinginequalities: d i · min ( 1 L L X s=1 r 2 x iH i (x i ,x i ,! s )) 1 L L X s=1 d i · ii (! s ) > 1 L L X s=1 X j6=i d j · ij (! s ) = 1 L L X s=1 X j6=i d j ·sup x2 X kr x i x jH i (x i ,x i ,! s )k = X j6=i d j · 1 L L X s=1 sup x2 X kr x i x jH i (x i ,x i ,! s )k ! X j6=i d j · sup x2 X 1 L L X s=1 kr x i x jH i (x i ,x i ,! s )k ! X j6=i d j · sup x2 X k 1 L L X s=1 r x i x jH i (x i ,x i ,! s )k ! = X j6=i d j · sup x2 X kr x i x j( 1 L L X s=1 H i (x i ,x i ,! s ))k ! . Asaresult,wehave: d i · min ( 1 L L P s=1 r 2 x i H i (x i ,x i ,! s )) P j6=i d j · ✓ sup x2 X kr x i x j( 1 L L P s=1 H i (x i ,x i ,! s ))k ◆ . Since the x we are considering here is selected arbitrarily from the set X, the above inequality holdsforanyx2X. Now,arbitrarilyselectN 1pointsfromX,e.g. y 1 ,y 2 ,...,y j6=i ,..,y N 2X. Thenwehave: d i · min ( 1 L L P s=1 r 2 x i H i (x i ,x i ,! s )) P j6=i d j ·k 1 L L P s=1 r y i j y j j H i (y i j ,y i j ,! s )k. In the above inequality, passing the limit ofL and using the continuity of min (·) and the equality (4.2),wehave: 104 d i · min (E[r 2 x i H i (x i ,x i ,!)]) P j6=i d j ·kE[r y i j y j j H i (y i j ,y i j ,!)]k ) d i · min (r 2 x i E[H i (x i ,x i ,!)]) P j6=i d j ·kr y i j y j j E[H i (y i j ,y i j ,!)]k ) d i · inf x2 X min (r 2 x i ✓ i (x i ,x i )) P j6=i d j · sup y j 2 X kr y i j y j j ✓ i (y i j ,y i j )k ) d i · ii P j6=i d j · ij foreachi. Hence,theassumption(A2)holds. ⇤ By now, we have already formulated the game model considered in this chapter and provided the assumptions of the ⇢ method. We will present the details of the ⇢ method in the following section. 4.2 Descriptionofthe⇢ Method Inthe⇢ method,wehavetwoloops. Ineachouteriterationmarkedasv,wesubstitutetheexpected- valuefunctionsineachplayer’sobjectiveandconstraintsbythesampledaveragefunctionsrespec- tively. However, those replacements may make some player’s sub-problem infeasible. To address it, in each playeri’s constraints, we add a non-negative variable ⇢ i on the RHS of each inequality whose LHS is a sampled average function. This adjustment can guarantee each player’s sub- problemisfeasible. However,sinceforeachi,⇢ i isunbounded,ifwedonothaveanypenalization of ⇢ i on player i’s objective, those constraints involving sampled average functions will not have any influence on i’s sub-problem at all. Hence, in each player i’s objective, we add a penalizing termof⇢ i . Under each outer iteration, the inner-loop is developed to apply the best-response algorithm to 105 solveadeterministicgame. Now,let’slookatthedescriptionofthe⇢ methodindetail. Algorithm2: ⇢ Method Let (L v ) 1 v=0 beapositivenon-decreasingsequencewithL 0 =0and lim v!1 L v =1. Foreachplayeri,undereachouteriterationv,define ˜ H i v (x i ,x i ), 1 Lv Lv P s=1 H i (x i ,x i ,! s ). Arbitrarily select an initial point (x 0,i ,⇢ 0,i ) N i=1 withx 0,i 2X i and ⇢ 0,i 0 fori=1,...,N and set v=1. (1) Newly generate the number ofL v L v 1 i.i.d. random samples of !. Then in the whole im- plementation,wehavealreadygeneratedthenumberofL v i.i.d. randomsamples,e.g. (! s ) Lv s=1 . Setµ=0andforeachplayeri,letx 0,v,i =x v 1,i . (2)Fori=1,...,N,define (x µ+1,v,i ,⇢ µ+1,v,i )asfollows: (x µ+1,v,i ,⇢ µ+1,v,i ), arg min (x i ,⇢ i )2 Z v,i ˜ H i v (x i ,x µ,v, i )+L v ⇢ i whereZ v,i , {(x i ,⇢ i )2X i ⇥ R + | 1 Lv Lv P s=1 G i j (x i ,! s ) ⇢ i 8j=1,2,...,k i }. (3) If some stopping criterion is satisfied, for each player i, let x v,i = x µ+1,v,i and proceed to the nextstep. Otherwise,updateµ µ+1andreturnto(2). (4) If some stopping criterion is satisfied, we terminate this algorithm and take the tuple (x v,i ) N i=1 asthecomputedN.E.ofthegame(4.1). Otherwise,updatev v+1andreturnto(1). From the description of the ⇢ method, we can see that in each outer iterationv, the inner-loop de- 106 scribedbysteps(1),(2)and(3)istosolvethefollowingdeterministicgame: 8 > > > < > > > : min x i 2 X i ,⇢ i 0 1 Lv Lv P s=1 H i (x i ,x i ,! s )+L v ⇢ i s.t. 1 Lv Lv P s=1 G i (x i ,! s ) ⇢ i 1 9 > > > = > > > ; N i=1 . (4.3v) For each v, define Z v , N Q i=1 Z v,i where Z v,i , {(x i ,⇢ i ) 2X i ⇥ R + | 1 Lv Lv P s=1 G i (x i ,! s )  ⇢ i 1} for any i =1,···,N. A Nash Equilibrium (N.E.) of the game (4.3v) is a tuple (x ⇤ ,v ,⇢ ⇤ ,v ) , (x ⇤ ,v,i ,⇢ ⇤ ,v,i ) N i=1 2Z v such that (x ⇤ ,v,i ,⇢ ⇤ ,v,i )2 argmin (x i ,⇢ i )2 Z v,i 1 Lv Lv P s=1 H i (x i ,x ⇤ ,v, i ,! s )+L v ⇢ i for any i=1,2,...,N. In the next section, we will show that under each outer iteration v, the game (4.3v) has a unique N.E.andtheinner-loopsolutionssequence,e.g. ((x µ,v,i ,⇢ µ,v,i ) N i=1 ) 1 µ=1 ,indeedconvergestoit. 4.3 Inner-LoopConvergenceAnalysis In the ⇢ method, under each outer iterationv, to solve the game (4.3v), we use the best-response algorithmlikewhathasbeenshowninthestep(2)inthedescription. When we look at the step (2), one immediate question will be asked. That is whether in each inner-iteration µ, for each player i, the solution (x µ+1,v,i ,⇢ µ+1,v,i ) can be well-defined. Since the objectivefunction 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s )+L v ⇢ i isnotstronglyconvexin⇢ i ,theexistenceofthe optimalsolution (x µ+1,v,i ,⇢ µ+1,v,i )isproblemistic. In4.3.1,wewillshowthatundereachouteriterationv,foranyinneriterationµ,givenanyx µ,v, i 2 X i ,theoptimizationproblem: min (x i ,⇢ i )2 Z v,i 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s )+L v ⇢ i hasauniqueoptimalsolutiondenotedby (x µ+1,v,i ,⇢ µ+1,v,i ). 107 4.3.1 TheExistenceandUniquenessofInnerIteration’sSolution In this sub-section, we will show that under each outer iteration v, for any inner iteration µ, the point (x µ+1,v,i ,⇢ µ+1,v,i )definedinthestep(2)ofthedescriptioniswell-defined. Specifically,toprovethat,wedevelopthefollowinglemma. Lemma9 Supposeassumptions(B1.1)–(B1.3)hold. Theninthe⇢ method,foranyouteriteration v 1 and inner iteration µ 0, given (x µ,v,i ,⇢ µ,v,i ) N i=1 with x µ,v,i 2X i and ⇢ µ,v,i 0 for each playeri,theoptimizationproblemofeachi presentedinthestep(2): min (x i ,⇢ i )2 Z v,i 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s )+L v ⇢ i (4.4) hasauniquesolution (x µ+1,v,i ,⇢ µ+1,v,i ) with⇢ µ+1,v,i =max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x µ+1,v,i ,! s )). Proof: Let’s fix an outer iterationv and an inner iterationµ. Then given (x µ,v,i ,⇢ µ,v,i ) N i=1 withx µ,v,i 2X i and ⇢ µ,v,i 0 for any i=1,2,...,N, for each player i, the function 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s ) is stronglyconvexinx i 2X i basedonassumption(B1.2). Hence,theoptimizationproblem: min x i 2 X i 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s )+L v max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x i ,! s )) (4.5) hasauniquesolution,e.g. ¯ x µ+1,v,i ,foranyi. While,fortheoptimizationproblem(4.4)formulated inthelemma’sstatement,wemusthave: ⇢ i max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x i ,! s )foranyfeasiblesolution (x i ,⇢ i )2Z v,i bythedefinitonofZ v,i . Therefore,foranyfeasiblesolution (x i ,⇢ i )2Z v,i ,wecandeducethat: 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s )+L v ⇢ i 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s )+L v max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x i ,! s )) 1 Lv Lv P s=1 H i (¯ x µ+1,v,i ,x µ,v, i ,! s )+L v max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (¯ x µ+1,v,i ,! s )). 108 Thelastinequalitycomesfromthefactthat (x i ,⇢ i )2Z v,i ) x i 2X i . Consequently, we have OPT 1 OPT 2 where OPT 1 , inf (x i ,⇢ i )2 Z v,i 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s )+L v ⇢ i andOPT 2 istheoptimalobjectivevalueoftheproblem(4.5). Reversely,let ¯ ⇢ µ+1,v,i =max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (¯ x µ+1,v,i ,! s )). Thenwemusthave (¯ x µ+1,v,i , ¯ ⇢ µ+1,v,i )2Z v,i . Now, we can conclude OPT 2 OPT 1 and thus OPT 1 = OPT 2 . As a result, (¯ x µ+1,v,i , ¯ ⇢ µ+1,v,i ) is anoptimalsolutionoftheoptimizationproblem(4.4). Bynow,theexistenceoftheoptimalsolutionofproblem(4.4)hasbeenestablished. Toshowtheuniqueness,supposewehavetwodifferentsolutionsoftheproblem(4.4), e.g. (x µ+1,v,i ,⇢ µ+1,v,i )and(ˆ x µ+1,v,i , ˆ ⇢ µ+1,v,i ). Fromtheoptimalitiesofthetwopointsintheproblem(4.4),wemusthave: ⇢ µ+1,v,i =max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x µ+1,v,i ,! s ))and ˆ ⇢ µ+1,v,i =max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (ˆ x µ+1,v,i ,! s )). Thus (x µ+1,v,i ,⇢ µ+1,v,i )6=(ˆ x µ+1,v,i , ˆ ⇢ µ+1,v,i )) x µ+1,v,i 6=ˆ x µ+1,v,i . Also,thefollowingequalitieshold: 1 Lv Lv P s=1 H i (x µ+1,v,i ,x µ,v, i ,! s )+L v max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x µ+1,v,i ,! s )) = 1 Lv Lv P s=1 H i (ˆ x µ+1,v,i ,x µ,v, i ,! s )+L v max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (ˆ x µ+1,v,i ,! s )) =min (x i ,⇢ i )2 Z v,i 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s )+L v ⇢ i =min x i 2 X i 1 Lv Lv P s=1 H i (x i ,x µ,v, i ,! s )+L v max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x i ,! s )). ThelastequalitycomesfromtheconclusionthatOPT 1 = OPT 2 . As bothx µ+1,v,i and ˆ x µ+1,v,i are in the set X i , the above equalities imply thatx µ+1,v,i and ˆ x µ+1,v,i are two different optimal solutions of the problem (4.5) contradicting to the uniqueness of the 109 solutionin(4.5). Hence the problem (4.4)’s optimal solution exists and is unique. Also, letting (x µ+1,v,i ,⇢ µ+1,v,i ) denotethisuniqueoptimalsolutionintheproblem(4.4),fromtheoptimality,wemusthave: ⇢ µ+1,v,i =max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x µ+1,v,i ,! s )),whichcanconcludeourproof. ⇤ Now, from the lemma 5, we can guarantee that in the ⇢ method, under each outer iterationv and inneriterationµ,thepoint (x µ+1,v,i ,⇢ µ+1,v,i )iswell-definedforanyplayeri. 4.3.2 Inner-LoopConvergenceProof With the preparations made in the last sub-section, we can give the inner-loop convergence result undereachouteriterationv inthefollowingtheorem. Theorem8 Suppose the assumptions (B1.1) – (B1.4) and (B2) are satisfied. Then under each outeriterationv,thegame(4.3v)hasauniqueN.E.andtheinner-loopsolutionssequence ((x µ,v,i ,⇢ µ,v,i ) N i=1 ) 1 µ=1 obtained by applying the best-response algorithm on the game (4.3v) will convergetothisuniqueN.E.denotedby (x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 with ⇢ 1 ,v,i =max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x 1 ,v,i ,! s )) foreachplayeri. Proof: In this proof, firstly let’s show that the sequence ((x µ,v,i ,⇢ µ,v,i ) N i=1 ) 1 µ=1 will converge and the limit pointisoneN.E.ofthegame(4.3v). Andthen,wewillshowtheuniquenessoftheN.E.. Fixanouteriterationv. Define ˜ H i v (x i ,x i ), 1 Lv Lv P s=1 H i (x i ,x i ,! s ). Thenimpliedbyassumption (B1.2), we know that ˜ H i v (x i ,x i ) is strongly convex inx i 2X i given anyx i 2X i . Also, from theassumption(B1.3),wecandeducethat 1 Lv Lv P s=1 G i j (x i ,! s )isconvexinx i foralli’sandj’sindi- 110 catingthatZ v,i isaclosedconvexsetforeachplayeri. Fromthedefinitionsandthe⇢ methoddescription,ataninneriterationµ,wehave: (x µ+1,v,i ,⇢ µ+1,v,i ), arg min (x i ,⇢ i )2 Z v,i ˜ H i v (x i ,x µ,v, i )+L v ⇢ i . Sincetheoptimizationproblem: min (x i ,⇢ i )2 Z v,i ˜ H i v (x i ,x µ,v, i )+L v ⇢ i (4.6) isconvexandalsofromassumption(B1.2),theobjectivefunctionoftheproblem(4.6)isdifferen- tiable in x i and ⇢ i , for each player i, we can use the optimality first order condition in the above problemunderaninneriterationµ 1 0tohavethefollowinginequality: 0 B B @ x i x µ,v,i ⇢ i ⇢ µ,v,i 1 C C A T 0 B B @ r x i ˜ H i v (x µ,v,i ,x µ 1,v, i ) L v 1 C C A 0, 8(x i ,⇢ i )2Z v,i . (4.7) Whileusingtheoptimalityfirstorderconditionintheinneriterationµgivesusthefollowing: 0 B B @ x i x µ+1,v,i ⇢ i ⇢ µ+1,v,i 1 C C A T 0 B B @ r x i ˜ H i v (x µ+1,v,i ,x µ,v, i ) L v 1 C C A 0, 8(x i ,⇢ i )2Z v,i . (4.8) Since both (x µ+1,v,i ,⇢ µ+1,v,i ) and (x µ,v,i ,⇢ µ,v,i ) are in Z v,i , we can substitute (x i ,⇢ i ) in both the inequalities (4.7) and (4.8) by (x µ+1,v,i ,⇢ µ+1,v,i ) and (x µ,v,i ,⇢ µ,v,i ) respectively and then combine theresultedtwoinequalitiestoobtainthefollowinginequality: (x µ+1,v,i x µ,v,i ) T (r x i ˜ H i v (x µ+1,v,i ,x µ,v, i )r x i ˜ H i v (x µ,v,i ,x µ 1,v, i )) 0. (4.9) As ˜ H i v (x i ,x i ) , 1 Lv Lv P s=1 H i (x i ,x i ,! s ), from inequality (4.9), we can deduce the following in- equalities: (x µ+1,v,i x µ,v,i ) T (r x i ˜ H i v (x µ+1,v,i ,x µ,v, i )r x i ˜ H i v (x µ,v,i ,x µ 1,v, i )) 0 =) 111 (x µ+1,v,i x µ,v,i ) T ( 1 Lv Lv P s=1 r x iH i (x µ+1,v,i ,x µ,v, i ,! s ) 1 Lv Lv P s=1 r x iH i (x µ,v,i ,x µ 1,v, i ,! s )) 0 =) Lv P s=1 (x µ+1,v,i x µ,v,i ) T (r x iH i (x µ+1,v,i ,x µ,v, i ,! s )r x iH i (x µ,v,i ,x µ 1,v, i ,! s )) 0. (4.10) Foreachs,byapplyingthemean-valuetheoremontheterm: (x µ+1,v,i x µ,v,i ) T (r x iH i (x µ+1,v,i ,x µ,v, i ,! s )r x iH i (x µ,v,i ,x µ 1,v, i ,! s )), wehave: (x µ+1,v,i x µ,v,i ) T (r x iH i (x µ+1,v,i ,x µ,v, i ,! s )r x iH i (x µ,v,i ,x µ 1,v, i ,! s )= (x µ+1,v,i x µ,v,i ) T r 2 x i H i (y µ,v,i (! s ),! s )(x µ+1,v,i x µ,v,i ) +(x µ+1,v,i x µ,v,i ) T r 2 x i x j H i (y µ,v,i (! s ),! s )(x µ,v, i x µ 1,v, i ) wherey µ,v,i (! s ),t µ,v,i (! s )·(x µ+1,v,i ,x µ,v, i )+(1 t µ,v,i (! s ))·(x µ,v,i ,x µ 1,v, i )forsome t µ,v,i (! s )2(0,1). Here,weusethenotationsy µ,v,i (! s )andt µ,v,i (! s )toindicatethatboththevaluesofy µ,v,i andt µ,v,i dependonthevalueoftherealizedrandomsample! s . Basedonassumption(B2),wehavethat: ii (! s )kx µ+1,v,i x µ,v,i k 2  (x µ+1,v,i x µ,v,i ) T r 2 x i H i (y µ,v,i (! s ),! s )(x µ+1,v,i x µ,v,i ). Hence,frominequality(4.10),wecandeducethefollowinginequalities: Lv P s=1 (x µ+1,v,i x µ,v,i ) T (r x iH i (x µ+1,v,i ,x µ,v, i ,! s )r x iH i (x µ,v,i ,x µ 1,v, i ,! s )) 0 =) ( Lv P s=1 ii (! s ))k(x µ+1,v,i x µ,v,i )k 2 + Lv P s=1 (x µ+1,v,i x µ,v,i ) T ( P j6=i r 2 x i x j H i (y µ,v,i (! s ),! s ) (x µ,v,j x µ 1,v,j )) 0 =) ( Lv P s=1 ii (! s ))kx µ+1,v,i x µ,v,i k 2 k x µ+1,v,i x µ,v,i k Lv P s=1 P j6=i ( ij (! s )kx µ,v,j x µ 1,v,j k) =) ( Lv P s=1 ii (! s ))kx µ+1,v,i x µ,v,i k P j6=i ( Lv P s=1 ij (! s ))kx µ,v,j x µ 1,v,j k 112 =)kx µ+1,v,i x µ,v,i k P j6=i Lv P s=1 ij (! s ) Lv P s=1 ii (! s ) kx µ,v,j x µ 1,v,j k. (4.11) Concatenatingtheinequalities(4.11)forallplayers,weobtain: 0 B B B B B B @ kx µ+1,v,1 x µ,v,1 k . . . x µ+1,v,N x µ,v,N 1 C C C C C C A  ˆ ⌥ v 0 B B B B B B @ kx µ,v,1 x µ 1,v,1 k . . . x µ,v,N x µ 1,v,N 1 C C C C C C A (4.12) where ˆ ⌥ v = 0 B B B B B B B B B B @ 0 v 12 v 11 ··· v 1N v 11 v 21 v 22 0 ··· v 2N v 22 . . . . . . . . . . . . v N1 v NN ··· ··· 0 1 C C C C C C C C C C A with v ij , Lv P s=1 ij (! s )foralli’sandj’s. Letd, (d i ) N i=1 be the set of positive constants being defined in assumption (B2) from where we have: v ii d i > P j6=i v ij d j (4.13) 8i=1,2,3,...,N. Usingthepropertypresentedby(4.13)foreachplayeriandtheinequality(4.12)andrecallingthat thenorm|·| d ismonotonic,wehave: 0 B B B B B B @ kx µ+1,v,1 x µ,v,1 k . . . x µ+1,v,N x µ,v,N 1 C C C C C C A d  v · 0 B B B B B B @ kx µ,v,1 x µ 1,v,1 k . . . x µ,v,N x µ 1,v,N 1 C C C C C C A d 113 with v , ˆ ⌥ v d beingapositivevaluestrictlylessthan1. DefineD v 1 , 0 B B B B B B @ kx 1,v,1 x 0,v,1 k . . . x 1,v,N x 0,v,N 1 C C C C C C A d . Thenforanyµ 1,wehave: 0 B B B B B B @ kx µ+1,v,1 x µ,v,1 k . . . x µ+1,v,N x µ,v,N 1 C C C C C C A d  µ v D v 1 . (4.14) Defined m , max 1 i N d i . From(4.14),foreachµ 1,wehavethefollowinginequalities: (4.14) =) max i=1,2,...,N kx µ+1,v,i x µ,v,i k d i  µ v D v 1 =) kx µ+1,v,i x µ,v,i k d m  µ v D v 1 8i=1,2,...,N =)kx µ+1,v,i x µ,v,i k µ v D v 1 d m 8i=1,2,...,N. (4.15) Due to the fact that 0< v < 1, from (4.15), for each player i, it can be easily verified that the sequence (x µ,v,i ) 1 µ=1 is a Cauchy sequence inR n i -space with thel 2 -norm being the metric. Due to the completeness of theR n i -space, we can conclude that (x µ,v,i ) 1 µ=1 must converge. Also, asX i is aclosedsubsetinR n i ,thelimitpointof (x µ,v,i ) 1 µ=1 ,e.g. x 1 ,v,i ,willbeinX i aswell. Fromlemma5,weknowthatforeachi,atiterationµ 1,wehavethefollowingequality: ⇢ µ,v,i =max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x µ,v,i ,! s )). Hence using the continuity of G i (x i ,!) in x i 2X i implied by assumption (B1.3), the sequence 114 (x µ,v,i ,⇢ µ,v,i ) 1 µ=1 will converge for alli’s. Concatenating the convergence properties for alli’s, we concludethatthesequence((x µ,v,i ,⇢ µ,v,i ) N i=1 ) 1 µ=0 convergestoapointdenotedas(x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 withx 1 ,v,i 2X i and⇢ 1 ,v,i =max(0, max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x 1 ,v,i ,! s ))foranyi. For each player i, as the continuity ofr x i ˜ H i v (x i ,x i ) in x2X is implied by assumption (B1.2) and the inequality (4.7) holds for any µ, passing to the limit as µ!1 in (4.7) enables us to concludethatthelimitpoint (x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 isoneN.E.ofthegame(4.3v). ThelastthinginthisproofistoshowtheuniquenessoftheN.E.inthegame(4.3v). SupposetherearetwoN.E.sinthegame(4.3v),e.g. (x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 and(¯ x 1 ,v,i , ¯ ⇢ 1 ,v,i ) N i=1 . Then in both of the two points, for eachi, using the optimality first order conditions respectively, wehave: 0 B B @ x i x 1 ,v,i ⇢ i ⇢ 1 ,v,i 1 C C A T 0 B B @ r x i ˜ H i v (x 1 ,v,i ,x 1 ,v, i ) L v 1 C C A 0, 8(x i ,⇢ i )2Z v,i (4.16) and 0 B B @ x i ¯ x 1 ,v,i ⇢ i ¯ ⇢ 1 ,v,i 1 C C A T 0 B B @ r x i ˜ H i v (¯ x 1 ,v,i ,¯ x 1 ,v, i ) L v 1 C C A 0, 8(x i ,⇢ i )2Z v,i . (4.17) Foreachi,replacing(x i ,⇢ i )by(¯ x 1 ,v,i , ¯ ⇢ 1 ,v,i )and(x 1 ,v,i ,⇢ 1 ,v,i )in(4.16)and(4.17)respectively andthencombiningthemtogether,wehave: (x 1 ,v,i ¯ x 1 ,v,i ) T (r x i ˜ H i v (x 1 ,v,i ,x 1 ,v, i )r x i ˜ H i v (¯ x 1 ,v,i ,¯ x 1 ,v, i )) 0. (4.18) 115 Usingthesimilaranalysesabove,wecaneasilyobtainthefollowinginequality: 0 B B B B B B @ kx 1 ,v,1 ¯ x 1 ,v,1 k . . . x 1 ,v,N ¯ x 1 ,v,N 1 C C C C C C A d  v · 0 B B B B B B @ kx 1 ,v,1 ¯ x 1 ,v,1 k . . . x 1 ,v,N ¯ x 1 ,v,N 1 C C C C C C A d (4.19) wheredand v havebeendefinedabovewithdbeingapositivevectorand v beingapositivevalue strictlylessthan1. Thus,theinequality(4.19)impliesthat (x 1 ,v,i ) N i=1 =(¯ x 1 ,v,i ) N i=1 ,whichconcludesourproof. ⇤ The theorem 8 guarantees the inner-loop’s convergence in the ⇢ method. To conclude the whole convergence analysis in this chapter, we need to answer the question that as the outer iteration v’s going to1, whether in some probabilistic sense, the sequence ((x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 )) 1 v=1 will converge to one N.E. of the original game (4.1) where (x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 is the unique N.E. of the game(4.3v)foreachv. 4.4 Outer-LoopConvergenceAnalysis In the last section, we have shown that in the ⇢ method, under each outer iterationv, the solutions sequenceobtainedbyimplementingthebest-responsealgorithmintheinner-loopwillconvergeto theuniqueN.E.ofthegame(4.3v)denotedas (x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 . Inthissection,wewillshowthat the sequence ((x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 )) 1 v=1 will converge to the N.E. of the original game (4.1) in a.e. undersomemildassumptions. 116 As X is compact, the sequence ((x 1 ,v,i ) N i=1 ) 1 v=1 will be bounded. Naturally, we hope that any accumulationpointofthesequence((x 1 ,v,i ) N i=1 ) 1 v=1 willbeoneN.E.ofthegame(4.1)ina.e.. The firststeptoshowitistoprovethatthesequence((⇢ 1 ,v,i ) N i=1 ) 1 v=1 willconvergeto0ina.e. suchthat alltheaccumulationpointsof ((x 1 ,v,i ) N i=1 ) 1 v=1 willbefeasibleintheoriginalgame(4.1)ina.e.. Thefollowinglemmagivestheproofforthatpoint. Lemma10 Suppose assumptions (B1.1) and (B1.4) – (B1.6) hold. Then in the ⇢ method, the sequence ((⇢ 1 ,v,i ) N i=1 ) 1 v=1 will converge to0 in almost everywhere (a.e.) where (⇢ 1 ,v,i ) N i=1 is the ⇢ partoftheN.E.– (x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 ofthegame(4.3v)obtainedintheouteriterationv. Proof: Since from assumption (B1.5), we know that for each playeri, in probability 1, the following two equalitieshold: lim L!1 [sup x i 2 X i || 1 L L P s=1 G i (x i ,! s ) E[G i (x i ,!)]|| 2 ]=0, (4.20) lim L!1 [sup x2 X || 1 L L P s=1 H i (x i ,x i ,! s ) E[H i (x i ,x i ,!)]|| 2 ]=0. (4.21) The number of players, e.g. N, is finite (thus is countable). Hence the probability that the equali- ties(4.20)and(4.21)holdforalli’sequals1aswell. Toprovethelemma,wejustneedtoshowthatforanyrealizedrandomsequence,e.g. w=(! s ) 1 s=1 , making the equalities (4.20) and (4.21) hold for alli’s, the induced sequence ((⇢ 1 ,v,i ) N i=1 ) 1 v=1 will convergeto0. Now,letw=(! s ) 1 s=1 beafixedsequencesuchthatboth(4.20)and(4.21)holdforanyi. Sincew hasbeenfixed,theinducedouter-loopsolutionssequence ((x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 ) 1 v=1 isdeter- ministicastheresult. Suppose under w, the induced sequence ((⇢ 1 ,v,i ) N i=1 ) 1 v=1 does not converge to 0. Then there ex- 117 ists a positive value✏> 0 such that |(⇢ 1 ,v,i ) N i=1 | l1 >✏ in infinitely many times where|·| l1 is the l 1 -norm defined on the R N -space. Using the non-negativity of ⇢ 1 ,v,i for any i and v, we have max i=1,2,...,N ⇢ 1 ,v,i >✏ in infinitely many times. That implies there exists one player i, such that ⇢ 1 ,v,i >✏ in infinitely many times, which means there is a subsequence of (⇢ 1 ,v,i ) 1 v=1 , e.g. (⇢ 1 ,v,i ) v2  ,suchthat⇢ 1 ,v,i >✏foranyv2 . From assumption (B1.6), there exists a Slater’s point ˆ x i 2ri(X i ) with E[G i (ˆ x i ,!)] < 0. Since underthesequencew,boththeequalities(4.20)and(4.21)holdforalli’s,wemusthaveapositive integerN w such that 1 Lv Lv P s=1 G i (ˆ x i ,! s ) < 0 for anyv2 withv N w . Hence, (ˆ x i ,0)2Z v,i for anyv2 beinglargeenough,underwhichwehave: 1 L v Lv X s=1 H i (ˆ x i ,x 1 ,v, i ,! s ) 1 L v Lv X s=1 H i (x 1 ,v,i ,x 1 ,v, i ,! s )+L v ⇢ 1 ,v,i =) 1 L v Lv X s=1 H i (ˆ x i ,x 1 ,v, i ,! s )> 1 L v Lv X s=1 H i (x 1 ,v,i ,x 1 ,v, i ,! s )+L v ✏ =)| 1 L v Lv X s=1 H i (ˆ x i ,x 1 ,v, i ,! s ) 1 L v Lv X s=1 H i (x 1 ,v,i ,x 1 ,v, i ,! s )|>L v ✏ =)| 1 L v Lv X s=1 H i (ˆ x i ,x 1 ,v, i ,! s ) E[H i (ˆ x i ,x 1 ,v, i ,!)]|+|E[H i (ˆ x i ,x 1 ,v, i ,!)| +| 1 L v Lv X s=1 H i (x 1 ,v,i ,x 1 ,v, i ,!) E[H i (x 1 ,v,i ,x 1 ,v, i ,!)]|+|E[H i (x 1 ,v,i ,x 1 ,v, i ,!)]| >L v ✏. (4.22) From assumptions (B1.1) and (B1.4), both |E[H i (ˆ x i ,x 1 ,v, i ,!)| and |E[H i (x 1 ,v,i ,x 1 ,v, i ,!)]| areboundedabove. Also,fromassumption(B1.5),weknowthatboth| 1 Lv Lv P s=1 H i (ˆ x i ,x 1 ,v, i ,! s ) E[H i (ˆ x i ,x 1 ,v, i ,!)]| and | 1 Lv Lv P s=1 H i (x 1 ,v,i ,x 1 ,v, i ,! s ) E[H i (x 1 ,v,i ,x 1 ,v, i ,!)]| can be arbi- trarily small as long asv is large enough. Therefore, the LHS of the inequality (4.22) is bounded 118 above whenv is large enough. While the RHS of the inequality (4.22) is unbounded whenv2 and goes to1. Consequently, when we restrict our analysis on the subsequence (⇢ 1 ,v,i ) v2  , the inequality(4.22)willbeviolatedeventually. Hence,thesequence ((⇢ 1 ,v,i ) N i=1 ) 1 v=1 willconvergeto0inalmosteverwhere(a.e.). ⇤ The lemma 10 gives us the convergence property of the sequence ((⇢ 1 ,v,i ) N i=1 ) 1 v=1 . The last thing remained to establish the outer-loop convergence property is to show that there exists at least one N.E. of the original game (4.1) and the sequence ((x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 ) 1 v=1 will converge to one of theN.E.sina.e.. Theorem9 Suppose the assumption (B1) is satisfied. Then the sequence ((x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 ) 1 v=1 is bounded point-wise and any accumulation point of ((x 1 ,v,i ) N i=1 ) 1 v=1 is one N.E. of the original game(4.1)inalmosteverywhere(a.e.). What’s more, if the assumption (A2) or (B2) holds and at meanwhile, for eachi,r 2 x H i (x i ,x i ,!) satisfies the uniform law of large numbers overX, then the original game (4.1) has a unique N.E. andthewholesequence ((x 1 ,v,i ) N i=1 ) 1 v=1 willconvergetoitinalmosteverywhere(a.e.). Proof: Fromassumption(B1.1),weknowthat ((x 1 ,v,i ) N i=1 ) 1 v=1 mustbeboundedpoint-wise. Based on assumption (B1.5), we know that the probability of realizing a random sequence w = (! s ) 1 s=1 suchthatthefollowingtwoequalitiesholdforalli’sis1: lim L!1 [sup x i 2 X i || 1 L L P s=1 G i (x i ,! s ) E[G i (x i ,!)]|| 2 ]=0, (4.23) lim L!1 [sup x2 X || 1 L L P s=1 H i (x i ,x i ,! s ) E[H i (x i ,x i ,!)]|| 2 ]=0. (4.24) 119 From now, we will focus on one random sequence w making (4.23) and (4.24) hold for all i’s simultaneously. Sincew hasbeenfixed,theinducedouter-loopsolutionssequence ((x 1 ,v,i ,⇢ 1 ,v,i ) N i=1 ) 1 v=1 isdeter- ministic as the result. Also, from the proof in lemma 6, we know that ((⇢ 1 ,v,i ) N i=1 ) 1 v=1 converges to0. Let (x 1 ,i ) N i=1 be an accumulation point of ((x 1 ,v,i ) N i=1 ) 1 v=1 . Then based on the equality (4.23) and theconclusionoflemma6,wehave: E[G i (x 1 ,i ,!)] 0foralli’s. Hence,foreachplayeri,x 1 ,i isfeasibleinhis/hersub-problemintheoriginalgame(4.1). Now, suppose (x 1 ,i ) N i=1 is not a N.E. of the original game. Then there must exist a player i and a point x ⇤ ,i being feasible in i’s sub-problem in the game (4.1) such that E[H i (x ⇤ ,i ,x 1 , i ,!)] < E[H i (x 1 ,i ,x 1 , i ,!)]. From assumption (B1.6), we know that in player i’s expected-value constraints, there exists a Slater’s point ˆ x i 2ri(X i ) such that E[G i (ˆ x i ,!)] < 0. Using the convexity of the expectation function E[G i (x i ,!)] implied by assumption (B1.3), we have that for any positive integer n, E(G i ((1 1 n )x ⇤ ,i + 1 n ˆ x i ,!))< 0. DuetotheconvexityofX i ,wecanseethat(1 1 n )x ⇤ ,i + 1 n ˆ x i 2X i aswell. Thenfromtheequality(4.23),whenv islargeenough,wehave: 1 Lv Lv P s=1 G i ((1 1 n )x ⇤ ,i + 1 n ˆ x i ,! s )< 0andso ((1 1 n )x ⇤ ,i + 1 n ˆ x i ,0)2Z v,i . Thus,aslongasv islargeenough,wealwayshave: 1 L v Lv X s=1 H i ((1 1 n )x ⇤ ,i + 1 n ˆ x i ,x 1 ,v, i ,! s ) 1 L v Lv X s=1 H i (x 1 ,v,i ,x 1 ,v, i ,! s )+L v ⇢ 1 ,v,i ) 1 L v Lv X s=1 H i ((1 1 n )x ⇤ ,i + 1 n ˆ x i ,x 1 ,v, i ,! s ) 1 L v Lv X s=1 H i (x 1 ,v,i ,x 1 ,v, i ,! s ). (4.25) 120 Combiningtheequality(4.24)andtheinequality(4.25),wehave: E[H i ((1 1 n )x ⇤ ,i + 1 n ˆ x i ,x 1 , i ,!)] E[H i (x 1 ,i ,x 1 , i ,!)]. (4.26) Since (4.26) holds for any positive integer n, using the continuity of E[H i (x i ,x i ,!)] in x i , we have: E[H i (x ⇤ ,i ,x 1 , i ,!)] E[H i (x 1 ,i ,x 1 , i ,!)], whichcontradictstoourassumptionE[H i (x ⇤ ,i ,x 1 , i ,!)]<E[H i (x 1 ,i ,x 1 , i ,!)]. Hence,anyaccumulationpointof ((x 1 ,v,i ) N i=1 ) 1 v=1 isoneN.E.ofthegame(4.1)ina.e.. Besides,supposetheassumption(B2)holdsandforeachi,r 2 x H i (x i ,x i ,!)satisfiestheuniform law of large numbers over X. Then by lemma 8, we know that the assumption (A2) holds. Or equivalently, we can directly suppose the assumption (A2) holds. Then by referring to the con- clusion of lemma 1, we can easily verify that the N.E of the game (4.1) is unique. Let (x ⇤ ,i ) N i=1 denotethisuniqueN.E.. Usingtheconclusionabove,weknowthatina.e.,allaccumulationpoints of ((x 1 ,v,i ) N i=1 ) 1 v=1 must be (x ⇤ ,i ) N i=1 . Since ((x 1 ,v,i ) N i=1 ) 1 v=1 is bounded point-wise, the whole se- quence ((x 1 ,v,i ) N i=1 ) 1 v=1 willconvergeto (x ⇤ ,i ) N i=1 ina.e.. ⇤ Combining all the conclusions of lemma 9, lemma 10, theorem 8 and theorem 9, we have the followingsummarizations. Ifassumptions(B1)and(B2)hold,theninthe⇢ method,ineachouteriterationv,thecorrespond- ing inner-loop solutions sequence converges to the unique N.E. of the game (4.3v). Besides, any accumulation point of the sequence of outer-loop’s N.E.s is one N.E. of the original game (4.1) in a.e.. Additionally,iftheassumption(A2)holds,thegame(4.1)hasauniqueN.E.andthesequence ofouter-loop’sN.E.sconvergestoitina.e.. 121 5 NumericalResults In this chapter, we will carry out some simulations for both the Lagrangian scheme and the ⇢ method to obtain the numerical results. The purpose of this chapter is to help us have a better understandingofthenaturesofthosetwomethods,includingtheconvergenceproperties,models’ efficienciesandsoon. AllthesimulationsinthischapterareconductedusingtheMatlabsoftware ofCVX3.0beta(http://cvxr.com/cvx/). 5.1 GeneralModelDescriptionandReviewsoftheMethods In this section, we first give the general formulation of the model considered in this chapter. Then we review the schemas of the Lagrangian scheme and the ⇢ method separately. Lastly, we point outsomekeypointsinthesimulationsforbothalgorithms. 5.1.1 ModelDescription Now,let’sintroducetheframeworkofthegamemodelconsideredinthischapter. In each game in our simulations, we have N players and each player i’s strategy is x i which is a l-dimensionalvector. The objective function of player i has the expected-value and is strongly convex in his/her own variablex i givenotherplayers’strategiesx i . LetD be an⇥ n matrix. LetD ij denote the element located at thei-th row and thej-th column of D. Also, let D(r 1 : r 2 ,c 1 : c 2 ) represent the sub-matrix of D composed by the rows whose indices are fromr 1 tor 2 and the columns whose indices are fromc 1 toc 2 with 0 r 1 <r 2  n and 0 c 1 <c 2  n. 122 Thentosetuptheobjectivefunctionsforallplayers,wehavethefollowing5steps. Step 1: We randomly generate alN⇥ lN matrixQ with each element having the distribution as U(0.5,1.5). Step2: Foreachi=1,2,...,N,wemodifythei-thblocksub-matrixofQ,e.g.Q(s i1 :s i2 ,s i1 :s i2 ) withs i1 = l⇥ (i 1) + 1 ands i2 = l⇥ i, to make it symmetric by replacing its lower triangular elementswithitsdiagonallysymmetricallyuppertriangularelementsrespectively. Step 3: For each diagonal element of matrix Q, e.g. Q ii for i =1,2,...,lN ⇥ lN, we let Q ii = P j6=i Q ij +Q ji 2 +1. Step4: Foreachplayeri=1,2,...,N,wemanuallygenerateal-dimensionalvectorq i . Step 5: Define ˜ Q , E[Q +Q R ] where Q R is a random matrix with E[Q R ij ]=0 for any i,j = 1,2,...,lN⇥ lN. For each player i=1,2,...,N, we define ˜ q i , E[q i +q i,R ] with q i,R being a randomvectorandE[q i,R ]= 0. Thenforeachi,therandomfunctioninhis/herobjectiveis: H i (x i ,x i ,!), x i T ·(Q+Q R )(s i1 :s i2 ,s i1 :s i2 )·x i + P j6=i x i T ·(Q+Q R )(s i1 :s i2 ,s j1 :s j2 )·x j +(q i +q i,R ) T ·x i wheres i1 ands i2 are defined the same with those in step 2 ands j1 = l⇥ (j 1)+1,s j2 = l⇥ j foreachj6=i. Thus,whenwetakeexpectationonH i (x i ,x i ,!),theplayeri’sobjectivefunctionwillbe: ✓ i (x i ,x i ),E[H i (x i ,x i ,!)] = x i T · ˜ Q(s i1 :s i2 ,s i1 :s i2 )·x i + P j6=i x i T · ˜ Q(s i1 :s i2 ,s j1 :s j2 )·x j +(˜ q i ) T ·x i . Followingthegeneralsettings,let’stalkabouttheassumptionsinthetwomethods. Based on steps 1, 2, and 3, we know that the Q’s block sub-matrix Q(s i1 : s i2 ,s i1 : s i2 ) is sym- metric and strictly diagonally dominant with all elements being positive. Hence, by following the proposition 2.2.20 in [8], we are assured that for each player i, ✓ i (x i ,x i ) is strongly convex in 123 x i given all other players’ strategies x i . Also, in the Lagrangian scheme, like what the remark 1 has mentioned, under the assumption (A1), the assumption (A3) is implied by the assumption ( ¯ A3). Here,itiseasilyverifiedthattheassumption( ¯ A3)holdsbynotingthataasymmetricmatrix ispositivesemi-definite(psd)iffitssymmetricpartispsd. Hence,inthesimulations,wejustneedtochecktheassumptions(A1)and(A2)intheLagrangian schemeandtheassumptions(B1)and(B2)inthe⇢ methodrespectively. For each player i, his/her constraints are composed by two parts. The first part is deterministic, compact and convex, e.g. 0  x i  CAP i , where CAP i is a l-dimensional positive vector. The secondpartofplayeri’sconstraintsisstochasticandcanbeformulatedasE[G i (x i ,!)] 0where ! is a m-dimensional random vector commonly shared by all players and for each realized !, G i (x i ,!)isak i -dimensionalvector-valuedfunctionthatisconvexinx i . Based on the above descriptions, the general game model used in this chapter can be mathemati- callyformulatedasbelow: 8 > > < > > : min x i 2 X i ✓ i (x i ,x i ) s.t. E[G i (x i ,!)] 0 9 > > = > > ; N i=1 (5.1) where for each player i, ✓ i (x i ,x i ) is defined in the step 5 above and X i represents the compact convexregion {x i 2R l |0 x i  CAP i }. Next,wewillhavethebriefreviewsforboththeLagrangianschemeandthe⇢ method. 124 5.1.2 ReviewoftheLagrangianScheme ThedescriptionoftheLagrangianschemeinchapter2aregeneral. Herewewillpresentthedetails inamorespecificway. Let’swritethespecificdetailsoftheimplementationinLagrangianschemeatfirst. ImplementationoftheLagrangianScheme For each player i, under each outer iteration v, in each inner iteration µ, define ˜ H i µ,v (x i ,x i ) , 1 Lµ Lµ P s=1 H i (x i ,x i ,! s ). Fix a positive value c and arbitrarily select an initial point (˜ x 0,i , ˜ 0,i ) N i=1 with ˜ x 0,i 2X i and ˜ 0,i 2R k i + for i=1,...,N. Select a large enough number ¯ L as the increasing step-size of the i.i.d. random samples used in each inner iteration. Select the tolerable consecutive differences usedinthestopingcriteriatoterminateouterandinner-loopsrespectively,e.g. ✏ O and✏ I . Startingfromthefirstouteriterationv=1,wehavethefollowingsteps. (1)Setµ=0andlet (x 0,v,i , 0,v,i ) N i=1 =(˜ x v 1,i , ˜ v 1,i ) N i=1 . (2)SetL µ = ¯ Lifµ=0. Otherwise,setL µ =L µ 1 + ¯ L. Newlygenerate ¯ Li.i.d. randomsamples of !, e.g. (! s ) Lµ s=Lµ ¯ L+1 , and thus in the outer iterationv, we have already generatedL µ i.i.d. ran- domsamples,e.g. (! s ) Lµ s=1 . Fori=1,...,N,define (x µ+1,v,i , µ+1,v,i )asbelow: µ+1,v,i , argmax i 0 min x i 2 X i ˜ H i µ,v (x i ,x µ,v, i )+( i ) T 1 Lµ Lµ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 c 2 || i ˜ v 1,i || 2 with x µ+1,v,i , arg min x i 2 X i ˜ H i µ,v (x i ,x µ,v, i )+( µ+1,v,i ) T 1 Lµ Lµ P s=1 G i (x i ,! s )+ c 2 ||x i ˜ x v 1,i || 2 125 c 2 || µ+1,v,i ˜ v 1,i || 2 . (3) If µ=0, update µ µ+1 and return to (2). Else ifµ> 0, when ||(x µ+1,v,i , µ+1,v,i ) (x µ,v,i , µ,v,i )|| 1  ✏ I for alli’s, proceed to step (4). Otherwise, updateµ µ+1 and return to (2). (4)Foreachi=1,...,N,assign ˜ x v,i =x µ+1,v,i and ˜ v,i = µ+1,v,i . (5)If||(˜ x v,i , ˜ v,i ) (˜ x v 1,i , ˜ v 1,i )|| 1  ✏ O foralli’s,weterminatethisalgorithmandtake(˜ x v,i ) N i=1 asanapproximationoftheN.E.intheoriginalgame. Otherwise,updatev v+1andreturnto(1). In the description above, we use the consecutive difference in l 1 -norm as the stopping criterion in both outer and inner-loops. Since we have shown the convergence properties in both outer and inner-loops in a.e., the small consecutive difference of the solutions sequence in the outer-loop or the inner-loop is a powerful indication of the sequential convergence. Hence, as long as the two barsofconsecutivedifferenceusedintheouterandinner-loopsstoppingcriteriaaresmallenough, theobtainedsolutioncouldbeexpectedtobeveryclosetotherealone. Also,fromtheabovedescription,wecanseethatintheinner-loop,whenµ=0,nomatterwhether theconsecutivedifferencebetween(x 0,v,i ) N i=1 and(x 1,v,i ) N i=1 satisfiestheinner-loopstoppingcrite- rion, we will continue the inner-loop implementation to the next inner iteration. The main reason is that the inner-loop in the Lagrangian scheme is stochastic and to ensure the solution we obtain isclosetothetheoreticalone,wealwayswanttohaveatleasttwoormoreinneriterationsineach inner-loopintheimplementation. Another thing to mention is that for each playeri, under eachµ andv, he/she has to solve a max- min problem in step (2). From the analysis in chapter 2, we can conclude that in the inner-loop, each max-min problem is equivalent to a min-max problem under the same Lagrangian function. 126 Usingthisequivalence,theproblemcanbetransformedtoa min-maxproblemwhereforplayeri, givenx i ,wecansolvetheoptimalsolutionof i parameterizedbyx i inthemaxproblemof i and thensolvetheoptimalsolutionofx i inthe minproblemofx i . Oncewehaveobtainedthesolution ofx µ+1,v,i ,wecanbringitbacktocomputetheoptimalsolutionof µ+1,v,i . Furthermore,inourimplementation,wearemoreinterestedinthexpartofthesolutionsandcom- pared to the value range of the x part, the value range of part could be much larger due to the unboundedness. Then in designing our implementation, in the outer-loop and each inner-loop, we can set different bars of the consecutive difference forx and parts respectively, which are used inthestoppingcriteria. Themodifieddescriptionofthesimulationbasedontheaboveideasarepresentedasbelow. ImplementationoftheLagrangianScheme For each player i, under each outer iteration v, in each inner iteration µ, define ˜ H i µ,v (x i ,x i ) , 1 Lµ Lµ P s=1 H i (x i ,x i ,! s ). Fix a positive value c and arbitrarily select an initial point (˜ x 0,i , ˜ 0,i ) N i=1 with ˜ x 0,i 2X i and ˜ 0,i 2R k i + for i=1,...,N. Select a large enough number ¯ L as the increasing step-size of the i.i.d. random samples used in each inner iteration. Select the tolerable consecutive differences for bothx and parts in the stoping criteria to terminate outer and inner-loops respectively, e.g. ✏ O,x , ✏ O, ,✏ I,x and✏ I, . Startingfromthefirstouteriterationv=1,wehavethefollowingsteps. (1)Setµ=0andlet (x 0,v,i , 0,v,i ) N i=1 =(˜ x v 1,i , ˜ v 1,i ) N i=1 . 127 (2)SetL µ = ¯ Lifµ=0. Otherwise,setL µ =L µ 1 + ¯ L. Newlygenerate ¯ Li.i.d. randomsamples of !, e.g. (! s ) Lµ s=Lµ ¯ L+1 . Combining with the previously generated samples in the outer iteration v,wehaveL µ i.i.drandomsamplesnow,e.g. (! s ) Lµ s=1 . Fori=1,...,N,define (x µ+1,v,i , µ+1,v,i )asfollows: µ+1,v,i j (x i ), argmax j 0 j Lµ · Lµ P s=1 G i j (x i ,! s ) c 2 ( j ˜ v 1,i j ) 2 forj=1,2,3,...,k i . x µ+1,v,i , arg min x i 2 X i ˜ H i µ,v (x i ,x µ,v, i )+ c 2 ·||x i ˜ x v 1,i || 2 + k i P j=1 µ+1,v,i j (x i ) Lµ · Lµ P s=1 G i j (x i ,! s ) c 2 ·k µ+1,v,i (x i ) ˜ v 1,i k 2 . Let µ+1,v,i j = µ+1,v,i j (x µ+1,v,i )forj=1,2,3,...,k i . (3) Ifµ=0, updateµ µ+1 and return to (2). Else ifµ> 0, when ||x µ+1,v,i x µ,v,i || 1  ✏ I,x and || µ+1,v,i µ,v,i || 1  ✏ I, for alli’s, proceed to the step (4). Otherwise, updateµ µ+1 andreturnto(2). (4)Fori=1,...,N,assign ˜ x v,i =x µ+1,v,i and ˜ v,i = µ+1,v,i . (5) If ||˜ x v,i ˜ x v 1,i || 1  ✏ O,x and || ˜ v,i ˜ v 1,i || 1  ✏ O, for all i’s, we terminate this algo- rithm and take (˜ x v,i ) N i=1 as an approximation of the N.E. in the original game. Otherwise, update v v+1andreturnto(1). 5.1.3 Reviewofthe⇢ Method Inchapter4,tosolvethestochasticgameswithexpected-valueconstraints,weintroducedanother methodnamedthe"⇢ Method". Thespecificdescriptionoftheimplementationinthismethodare 128 shownbelow. Implementationofthe⇢ Method Foreachplayeri,undereachouteriterationv,define ˜ H i v (x i ,x i ), 1 Lv Lv P s=1 H i (x i ,x i ,! s ). Arbitrarilyselectaninitialpoint (x 0,i ,⇢ 0,i ) N i=1 withx 0,i 2X i and⇢ 0,i 0fori=1,...,N. Selecta large enough number ¯ L as the increasing step-size of the i.i.d. random samples used in each outer iteration. Select the tolerable consecutive differences in the stoping criteria to terminate outer and inner-loopsrespectively,e.g. ✏ O and✏ I . Startingfromthefirstouteriterationv=1andlettingL 0 =0,wehavethefollowingsteps. (1) Setµ=0 and letL v = L v 1 + ¯ L. Generate ¯ L i.i.d. random samples of !, e.g. (! s ) Lv s=Lv ¯ L+1 . Combining with the previously generated samples in the whole implementation, we haveL v i.i.d randomsamplesnow,e.g. (! s ) Lv s=1 . Foreachplayeri,letx 0,v,i =x v 1,i and⇢ 0,v,i = ⇢ v 1,i . (2)Fori=1,...,N,define (x µ+1,v,i ,⇢ µ+1,v,i )asfollows: (x µ+1,v,i ,⇢ µ+1,v,i ), arg min (x i ,⇢ i )2 Z v,i ˜ H i v (x i ,x µ,v, i )+L v ⇢ i whereZ v,i , {(x i ,⇢ i )2X i ⇥ R + | 1 Lv Lv P s=1 G i j (x i ,! s ) ⇢ i 8j=1,2,...,k i }. (3) If µ=0, update µ µ+1 and return to (2). Else ifµ> 0, when ||(x µ+1,v,i ,⇢ µ+1,v,i ) (x µ,v,i ,⇢ µ,v,i )|| 1  ✏ I for alli’s, proceed to step (4). Otherwise, updateµ µ+1 and return to (2). (4)Fori=1,...,N,assignx v,i =x µ+1,v,i and⇢ v,i = ⇢ µ+1,v,i . (5)Ifv=1,updatev v+1andreturnto(1). Elseifv> 1,when||(x v,i ,⇢ v,i ) (x v 1,i ,⇢ v 1,i )|| 1 129  ✏ O for alli’s, we terminate this algorithm and take (x v,i ) N i=1 as an approximation of the N.E. in theoriginalgame. Otherwise,updatev v+1andreturnto(1). In the ⇢ method, as the outer-loop is stochastic, we will continue our implementation regardless of the consecutive difference in the first outer iteration. Also, as the errors of the inner-loops’ so- lutions have the accumulative effects on the final solution error, we want to have a more accurate solution obtained in each inner-loop and hence we will continue our inner-loop implementation whenµ=0aswell. Again, in the ⇢ method, we use the consecutive difference inl 1 -norm as the stopping criterion in boththeouterandinner-loops. In the above description, in steps (3) and (5), similar to what we have done in the Lagrangian scheme,wecanalsosetdifferentbarsoftheconsecutivedifferencesinthestoppingcriteriaforthe xpartandthe⇢ partinboththeouterandinner-loops. However,wefindthismodificationdoesnot bring any obvious improvement in time efficiency because in the real implementation, the ⇢ part willconvergeto0veryfastduetothelargevalueofL v (recallingthatineachouteriterationv,for eachi,⇢ i ispenalizedonhis/herobjectivefunctionwiththepenalizationcoefficientbeingL v ). In the ⇢ method, what we should pay more attention is the satisfication of the assumption (B2). In (B2), the DD condition must be satisfied under any sample, which means when the random variablesarecontinuous,itisimpossibletoverifytheDDconditionunderanyscenario. However, ineachimplementation,toensuretheDDconditionissatisfied,wejustneedtocheckitunderthe mostextremescenario. Thefollowinglemmaprovesthispointformally. Lemma11 SupposeA andB are twon⇥ n real matrices wheren is a positive integer and also 130 assume 0| A ij | B ij for all i,j =1,2,...,n. Then we havekAk 2 k Bk 2 wherek·k 2 representstheinducedl 2 -normforanyn⇥ n matrix. Proof: Basedonthedefinitionoftheinducedmatrixnorm,weknowthatforanyn⇥ nmatrixD,wehave kDk 2 , max kxk 2 =1 kDxk 2 . As the feasible set {x2R n |kxk 2 =1} is compact, there must exist one vectorx ⇤ ,D 2R n suchthatkx ⇤ ,D k 2 =1andkDk 2 =kDx ⇤ ,D k 2 . ForthematrixA,letx ⇤ ,A 2R n beavectorwithkx ⇤ ,A k 2 =1andkAk 2 =kAx ⇤ ,A k 2 andy ⇤ ,A 2R n bethevectorsuchthaty ⇤ ,A i = |x ⇤ ,A i |foranyi=1,2,...,n. Thenky ⇤ ,A k 2 =1aswell. Now, for each indexi=1,2,...,n, under the assumptions in the lemma’s statement, we have the followingdeductions: |(Ax ⇤ ,A ) i | = | n P j=1 A ij x ⇤ ,A j | n P j=1 |A ij x ⇤ ,A j | = n P j=1 |A ij |·|x ⇤ ,A j | n P j=1 B ij y ⇤ ,A j =(By ⇤ ,A ) i , which imply thatkAk 2 =kAx ⇤ ,A k 2 k By ⇤ ,A k 2 k Bk 2 where the first inequality comes from themonotonicityofthel 2 -norm. ⇤ Remark 10: There may be a more general result than what the lemma says. However, as the conclusion of this lemma is already enough to help us defend our simulations, we introduce the lemmaandgiveitsproofherespecifically. Now,withthosepreparationsinthissection,itisthetimetoprovidethesimulations’resultsunder differentcasesinthetwomethods. 131 5.2 Case1: 5⇥ 5⇥ 2 Inthissection,wewillshowthesimulations’resultsforbothLagrangianschemeandthe⇢ method under the model having 5 players each of whom has 5 decision variables. There are two different linear expected-value constraints in each player’s sub-problem and hence we name the model as “5⇥ 5⇥ 2". The general structure of this model follows the formulation described in section 5.1, whilethespecificformulationofitwillbegivenbelow. 5.2.1 ModelIntroductionandGeneration Following the general formulation given in section 5.1, the objective function of each player i, e.g. ✓ i (x i ,x i ), has the quadratic form in this sub-section. For each i, let s i1 =5⇤ (i 1) + 1 and s i2 =5⇤ i. Then define ✓ i (x i ,x i ) , x i T · ˜ Q(s i1 : s i2 ,s i1 : s i2 ) · x i + P j6=i x i T · ˜ Q(s i1 : s i2 ,s j1 : s j2 )·x j +(˜ q i ) T ·x i where ˜ Q = E[Q +Q R ] withQ being a 25⇥ 25 deterministic matrix generated strictly following the instruction in 5.1.1. The random matrix Q R also has the 25⇥ 25 dimensionwithzeromeanforeachofitselement. Inthedefinitionof✓ i (x i ,x i ), the ˜ q i isdefined as ˜ q i ,E[q i +q i,R ]whereq i isa5-dimensionaldeterministicvectorselectedmanuallyandq i,R isa 5-dimensionalzero-meanrandomvector. Now,wehavethefollowing“5⇥ 5⇥ 2"modelpresented asbelow: 8 > > > > > > < > > > > > > : min x i 2 X i ✓ i (x i ,x i ) s.t. E[coef1(i,:)x i R1(i) b1(i)] 0 E[(coef2(i,:)+R2(i,:))x i b2(i)] 0 9 > > > > > > = > > > > > > ; 5 i=1 (5.2) 132 whereR1isa5-dimensionalrandomvectorandcoef1isa5⇥ 5matrixwitheachrowrepresenting the linear coefficients of the corresponding player’s decision variables in his/her first expected- value constraint. Also, coef2 and R2 are both 5⇥ 5 matrices. Each row of the sum of them representsthelinearcoefficientsofthecorrespondingplayer’sdecisionvariablesinhis/hersecond expected-value constraint. One thing to mention is that both coef1 and coef2 are constants in this model, while bothR1 andR2 are composed by the random elements. Besides,b1 andb2 are two 5-dimensionaldeterministicvectorsgeneratedmanually. Inthismodel,themethodtogeneratethedeterministicmatrixQhasbeenintroducedinsection5.1. Theothermodelelementscanbeclassifiedintotwogroups. Thefirstonecontainsallthemodel’s constantsincluding (CAP i ) 5 i=1 ,coef1,coef2,b1,b2and (q i ) 5 i=1 . Thesecondgroupcontainsallthe model’srandomelementscontainingR1,R2, (q i,R ) 5 i=1 andQ R . Themethodstogenerateallthemembersingroup1aregivenbelow. 1. For each i, CAP i is a 5-dimensional positive vector with each element being independently randomlygeneratedbycomputerwithdistributionU(1,1.3). 2. Each element in both coef1 and coef2 is also independently randomly generated by computer withdistributionU(0.6,1.4). 3. Theelementsinb1,b2and (q i ) 5 i=1 areallselectedmanuallytoguaranteetheexistencesofsome activeexpected-valueconstraints. In the second group, all members are random. Let’s introduce their elements’ distributions as be- low. 1. AllelementsinR1,R2and(q i,R ) 5 i=1 havethesamedistributionU( d,d)withdbeingapositive value. 2. AsforthematrixQ R ,foreachi,weletitsi-thdiagonalblocksub-matrixbea5⇥ 5zeromatrix, 133 e.g. Q R (s i1 : s i2 ,s i1 : s i2 )= 0 5⇥ 5 , where s i1 and s i2 are already defined before. For all other elementsinQ R ,weleteachofthemhavethesamedistributionU( d,d)aswell. Allthevaluesofrandomelementsinthismodelarerealizedindependentlyinourimplementations. Astherearesomanyconstantsdescribedinthismodel,itisimpossibletodemonstrateallofthem here. Wehavefoldedanduploadedthemonline. Thelinktoaccessthemis: http://suo.im/6g7vMi. Thenameofthecorrespondingfolderis"Simulationcase1: 5⇥ 5⇥ 2". Inthemodelofcase1,itcanbeeasilycheckedthattheassumption(A1)holds. Asfortheassump- tion(A2),wefirstcomputethevalueofthematrix ˆ ⌥ 5⇥ 5 with ˆ ⌥ ij = ij foralli,j=1,2,...,5and then upload ˆ ⌥ 5⇥ 5 online. Using the same link: http://suo.im/6g7vMi, people can access it. From thecomputedresult,wecanseethat ii > P j6=i ij foralli’s. Andthus,theassumption(A2)holds. Here,wemustnotethatthegamedescribedaboveisequivalenttoadeterministicone,whichmeans each expected-value function can be reduced to be deterministic. However, this point will not un- dermine the power to illustrate our algorithms’ properties in the simulation. Meanwhile, based on the proof in lemma 1, the satisfications of (A1) and (A2) enable us to use the best-response algorithm to solve the deterministic game directly to obtain the exact N.E. so that we can use this solution to check whether each of the two algorithms is able to give us a good estimation of the realN.E.. OnethingtohighlightisthatattheN.E.solution,thereareindeedsomeexpected-valueconstraints beingactive,whichmeansourmodelhereismeaningful. TheN.E.solutionofthismodelbeingcomputedbyapplyingthebest-responsealgorithmisshown intable1below. 134 x ⇤ ,1 x ⇤ ,2 x ⇤ ,3 x ⇤ ,4 x ⇤ ,5 0.3429 0.2676 0.1096 0.5108 0.3031 0.5422 0.7092 0 0 0.1286 0.4147 0.5155 0.4158 0.3970 0 0.2292 0.2497 0.2144 0 0.2146 0.4715 0 0.3019 0.5653 0.2775 Table1: N.E.solutionincase1 At the N.E. solution, the corresponding optimal dual values for each player i’s expected-value constraintsaresummarizedintable2below. ⇤ ,1 ⇤ ,2 ⇤ ,3 ⇤ ,4 ⇤ ,5 0 0 15.092 0 15.659 13.303 0 0 0 0 Table2: Dualsolutionincase1 From the optimal values of the dual variables, we can see that there are at least 3 active expected- valueconstraintsoverallplayersattheN.E.solution. Next,let’schecktheperformancesofthetwoalgorithms. 5.2.2 Case1: LagrangianScheme Now,let’sdemonstratethesimulationresultsincase1usingtheLagrangianscheme. Aswhathavebeenmentionedabove,incase1,themodelsatisfiesalltheassumptions(A1)-(A3). IntheLagrangianscheme,cisahyper-parameter. Thefirststepistotuneit. Todothat,weleteach random variable appeared in this case have the specific distributionU( 0.11,0.11), which means the variance of each random variable is about 0.004. Also, in each outer iteration, the increasing step-size of the i.i.d. random samples used in each inner iteration is 500. Furthermore, in case 1, 135 the outer-loop and inner-loops tolerable consecutive differences used in their stopping criteria are ✏ O,x =0.001,✏ O, =0.2,✏ I,x =0.0001and✏ I, =0.2respectively. In the convergence analysis, we know that as long as the assumptions (A1) to (A3) are satisfied, theconvergencepropertiesforbothouterandinner-loopscanbeprovedregardlessofthevalueof c. Hence,ccanbeanypositivevalue. Theoretically,impliedbytherateanalysis,whencislarger, the inner-loop convergence speed should be faster. Nevertheless, in the outer-loop, with a larger value of c, the difference between the games (2.6) and (2.15v) will be more obvious leading the number of outer iterations in the implementation to be larger. Hence, in the experiments, we tune thevalueofcwithintherangefrom0.00005to0.01, e.g. c2{ 0.00005,0.0001,0.0005,0.001,0.005,0.01}. Foreachvalueofc,wehavetheexperimentsfor10times. Underaspecificc,ineachexperiment, we have one initialized starting point, e.g. (˜ x 0,i , ˜ 0,i ) 5 i=1 . And the set of initialized starting points in the corresponding 10 experiments keeps the same over all different values of c’s. Each result shown below is computed by taking the average of the corresponding results in those 10 experi- ments. Now,themainstatisticsofthetuningresultsareshownintable3. 136 c total outer iterations total inner iterations running time random samples error 0.00005 3.4 52.9 504.73 26450 0.00036693 0.0001 2.5 32.7 301.76 16350 0.00040142 0.0005 3 37.8 348.65 18900 0.00041735 0.001 3 39.4 379.64 19700 0.00040206 0.005 4 48.2 439.43 24100 0.00048872 0.01 4 48.2 434 24100 0.00038199 Table3: C’stunningincase1 From the table above, as the increasing step-size of the i.i.d. random samples is fixed to be 500, the total number of random samples equals 500 times the number of total inner iterations. Also, the error fluctuates when c changes and all the errors are acceptable. What’s more, the average number of inner iterations in each outer iteration, e.g. total inner iterations / total outer iterations, doesnotchangeobviouslyindifferentc’s,exceptforthecasewherec=0.00005thatistoosmall to be counted. Considering that all the c’s are relatively small here, the above observation about theaveragenumberofinneriterationsisreasonable. Now,fromthetimeefficiency’spointofview,weselectc=0.0001fortheformalsimulation. In this simulation, we will mainly illustrate two things. The first one is whether the Lagrangian scheme will converge to a point close enough to the exact N.E. solution. The second one is how the variance of each random variable and the increasing step-size influences the simulation’s time efficiency. 137 Thevaluesofvarianceswewillinvestigateherearesummarizedinthefollowingset: {0.0001,0.0003,0.0007,0.001,0.004,0.007,0.01,0.03,0.05}. Under different model variances, the increasing step-sizes in each inner iteration are summarized intable4. variance Increasingstep-size ¯ L 0.0001 100 0.0003 100 0.0007 100 0.001 500 0.004 500 0.007 500 0.01 5000 0.03 5000 0.05 5000 Table4: Increasingstep-sizesunderdifferentvariancesincase1 Now,givenallofthosesettingsabove,theresultsofthesimulationareshownintable5. 138 variance total outer iterations total inner iterations running time random samples error 0.0001 2.4 16.2 158.21 1620 0.00019902 0.0003 2.2 21.7 209.09 2170 0.00034147 0.0007 2.4 35.8 348.23 3580 0.00039222 0.001 2.3 19.5 190.96 9750 0.00030447 0.004 2.7 36.7 359.64 18350 0.00044027 0.007 2.5 41.8 415.12 20900 0.00050415 0.01 2.3 18.8 187.74 94000 0.00024452 0.03 2.5 29.9 300.07 1.50E+05 0.0003097 0.05 2.4 36.4 363.76 1.82E+05 0.00042189 Table5: SimulationresultsoftheLagrangianschemewithc=0.0001incase1 For each value of the variance, we have the experiments for 10 times. Under a specific variance, ineachexperiment,wehaveoneinitializedstartingpoint,e.g. (˜ x 0,i , ˜ 0,i ) 5 i=1 . Thesetofinitialized starting points in the corresponding 10 experiments keeps the same over all different variances. Each result shown above is computed by taking the average of the corresponding results in those 10experiments. Fromthedatashownintheabovetable,wecanseethatfixing ¯ L,whenthevarianceincreases,the average number of inner iterations increases. While at the breaking points where we increase the ¯ L such as variance = 0.001 and variance = 0.01, the average numbers of inner iterations have the suddendecreasesduetothelarger ¯ L’sindicatingthatwhen ¯ Lislarger,theinner-loopwillconverge 139 fasterattheexpenseofgeneratingmorei.i.d. randomsamplesineachinneriteration. When looking at the data, we can see that given ¯ L, as the variance becomes larger, the time spent ontheimplementationincreasesandthenumberofgeneratedi.i.drandomsamplesbecomeslarger aswell. Generally, the above table shows that the Lagrangian scheme indeed works well and the time efficiencies of the implementations are good enough under all variances. Also, the Lagrangian scheme is able to help us obtain a very good approximation of the N.E. solution of the game we areconsideringhere. Next,let’slookattheperformancesofthe⇢ methodincase1. 5.2.3 Case1: ⇢ Method Inthe⇢ method,wealsocarryouttheimplementationsunderthesamesetofvariances,e.g. {0.0001,0.0003,0.0007,0.001,0.004,0.007,0.01,0.03,0.05}. In the simulation for this method, it is in the outer-loop, instead of the inner-loop, where we in- crease the size of generated i.i.d random samples. The selection rule of the increasing step-size according to different variances is the same with it used in the Lagrangian scheme and has been summarizedintable4. Beforeweprovidethesimulation’sresultsofthe⇢ method,weneedtoverifythatboththeassump- tions (B1) and (B2) hold. The verification of (B1) is routine, while the verification of (B2) is not straightforward. Toverifythat(B2)indeedholdsinthesimulationincase1,let’sfirstrecaptheassumption(B2)as below. 140 (B2): DiagonalDominance(DD)Condition: Foranyrealizedrandomsample! andeachplayeri,wedefine: (1) ii (!), inf x2 X smallesteigenvalueofr 2 x i H i (x i ,x i ,!)2(0,1), (2) ij (!), sup x2 X r 2 x i x j H i (x i ,x i ,!) 2 <1,forallj6=i; Thenthereexistsasetofpositiveconstants (d i ) N i=1 suchthatforanyrealized!, wehave: ii (!)d i > P j6=i ij (!)d j 8i=1,2,3,...,N. For each player i, according to the definitions, we have H i (x i ,x i ,!), x i T ·Q 0 ! (s i1 : s i2 ,s i1 : s i2 ) ·x i + P j6=i x i T ·Q 0 ! (s i1 : s i2 ,s j1 : s j2 ) ·x j +(q i +q i,R ! ) T ·x i where Q 0 ! , Q +Q R ! with Q R ! representing the value of the random matrix Q R under the realized random sample ! and all the s i1 , s i2 , s j1 and s j2 are defined the same with those in 5.5.1. The q i,R ! represents the value of the randomvectorq i,R undertherealizedrandomsample!. Only the random matrixQ 0 ! is directly related to (B2). Following the model generation steps de- scribed in 5.5.1, we know that for any i, Q 0 ! (s i1 : s i2 ,s i1 : s i2 ) is independent of ! and equals Q(s i1 :s i2 ,s i1 :s i2 ). Also,wehaveQ 0 ! (s i1 :s i2 ,s j1 :s j2 ) = Q(s i1 : s i2 ,s j1 : s j2 )+ Q R ! (s i1 : s i2 ,s j1 : s j2 ) for any j 6= i. Hence, in case 1, for each i, ii (!)= the smallest eigenvalue of 2⇤ Q(s i1 : s i2 ,s i1 : s i2 ) for any ! and ij (!)= kQ 0 ! (s i1 :s i2 ,s j1 :s j2 )k 2 = Q(s i1 :s i2 ,s j1 :s j2 )+Q R ! (s i1 :s i2 ,s j1 :s j2 ) 2 foranyj6=i. As in case 1, each random variable has the same distribution U( d,d) withd> 0, the largest valueofdcorrespondingtotheabove9differentvariancesequalsabout0.3873achievedwhenthe variance equals 0.05. However, in 5.2.1, we know, for any i and j 6= i, each element in the sub- matrix Q(s i1 : s i2 ,s j1 : s j2 ) has the value range of [0.5,1], which means the elements in Q(s i1 : 141 s i2 ,s j1 : s j2 )+Q R ! (s i1 : s i2 ,s j1 : s j2 ) must be positive in all scenarios in the simulation of case 1. Thenbyreferringtothelemma8,weknowthatunderanyscenarioinoursimulation,thevalue Q(s i1 :s i2 ,s j1 :s j2 )+Q R ! (s i1 :s i2 ,s j1 :s j2 ) 2 isboundedbykQ(s i1 :s i2 ,s j1 :s j2 )+0.3873k 2 for any i and j 6= i. Therefore, for each i and j 6= i, defining ¯ ii , the smallest eigenvalue of 2⇤ Q(s i1 : s i2 ,s i1 : s i2 ) and ¯ ij ,kQ(s i1 :s i2 ,s j1 :s j2 )+0.3873k 2 , if we can show there exists a set of positive values, e.g. (d 1 ,d 2 ,...,d N ), such that ¯ ii d i > P j6=i ¯ ij d j for alli’s, the assumption (B2)willbesatisfiedinallofourimplementationshere. Thematrix ¯ 5⇥ 5 with ¯ ij , ¯ ij foralli’sandj’scanbeaccessedviathefollowinglink: http://suo.im/6g7vMi. From the value of the matrix ¯ 5⇥ 5 , we can see that whend i =1 for anyi, the inequality ¯ ii d i > P j6=i ¯ ij d j holdsforalli’s,whichmeans(B2)holdsoveralltheimplementationsofthe⇢ methodin case1. Inthesimulation,weset✏ O =0.0001and✏ I =0.001. Nevertheless,intheLagrangianscheme,we set✏ O,x =0.001and✏ I,x =0.0001. Thereasontointerchangethevaluesbetween✏ O and✏ I inthe primal solution part is that in the Lagrangian scheme, the inner-loop is stochastic with the outer- loop being deterministic, while in the ⇢ method, in each outer iterationv, the inner-loop solves a deterministicgameandtheouter-loophereisactuallystochastic. Toobtainamoreaccurateresult, wealwayslettheloopwiththestochasticpropertyhasthestricterbaroftheconsecutivedifference inthestoppingcriterion. Thus,wemakesuchinterchangehere. Now, we can look at the performances of the ⇢ method under 9 different model variances in case 1. 142 variance total outer iterations total inner iterations running time random samples error 0.0001 5.4 13.4 36.371 540 0.00018018 0.0003 8.9 20.4 56.436 890 0.00026595 0.0007 12.8 28.2 75.822 1280 0.00040604 0.001 7.7 18 49.909 3850 0.00025935 0.004 12.7 28 74.932 6350 0.0003984 0.007 15.8 34.1 92.177 7900 0.0005185 0.01 7.1 16.8 47.142 35500 0.00028169 0.03 12 26.6 75.083 60000 0.00035602 0.05 15 32.6 91.258 75000 0.00040895 Table6: Simulationresultsofthe⇢ methodincase1 Again, in the ⇢ method, under each variance, we run the experiments for 10 times and each result shown above is computed by taking the average of the corresponding results in those 10 experi- ments. For each variance, the set of thex parts of the initial points used in the 10 experiments is thesamewithitintheLagrangianscheme. From the table above, we can see that compared to the Lagrangian scheme, the ⇢ method has the betterperformancesinboththerunningtimeandthenumberofgeneratedi.i.d. randomsamples. Like what we just discussed, in the ⇢ method, the outer-loop is stochastic, while the inner-loop is deterministic. Hence, the table 6 shows that compared with the Lagrangian scheme, under a fixed increasing step-size, the ⇢ method has the more unstable number of total outer iterations and the 143 morestablenumberofaverageinneriterationswhenthevariancechanges. 5.3 Case2: 8⇥ 8⇥ 2 In case 1, we have 5 players with each player having 5 decision variables and 2 expected-value constraints. In case 2, we will enlarge the model size. In this model, we have 8 players with each player having 8 decision variables and 2 expected-value constraints. We name this model as “8⇥ 8⇥ 2". 5.3.1 ModelIntroductionandGeneration Except for the model size, the details of the model used in case 2 are very similar to those in case 1. Here,weredefines i1 , 8⇤ (i 1)+1ands i2 , 8⇤ iforeachi=1,2,...,8. Allthematrices ˜ Q, Q andQ R have the dimension of 64⇥ 64. Also, correspondingly, for eachi, all the ˜ q i ,q i andq i,R are8-dimensionalvectors. Basedonthegeneralintroductionsin5.1.1,foreachi,wehavehis/her objectivefunctionas: ✓ i (x i ,x i ),x i T · ˜ Q(s i1 :s i2 ,s i1 :s i2 )·x i + P j6=i x i T · ˜ Q(s i1 :s i2 ,s j1 :s j2 )·x j +(˜ q i ) T ·x i . Now,wewillhavethespecific“8⇥ 8⇥ 2"gamemodelincase2asbelow: 8 > > > > > > < > > > > > > : min x i 2 X i ✓ i (x i ,x i ) s.t. E[coef1(i,:)x i R1(i) b1(i)] 0 E[(coef2(i,:)+R2(i,:))x i b2(i)] 0 9 > > > > > > = > > > > > > ; 8 i=1 . (5.3) The dimensions of all the above model components in each player’s constraints are also enlarged correspondinglytoadapttotheincreasedmodelsize. 144 Inthismodel,thegenerationmethodofQisthesamewithitincase1. Asforalltheothermodelelements,thegenerationmethodsaresummarizedbelow. 1. For each i, CAP i is a 8-dimensional positive vector with each element being independently randomlygeneratedbycomputerwithdistributionU(0.5,0.7). 2. Each element in both coef1 and coef2 is also independently randomly generated by computer withdistributionU(0.6,1.4). 3. The elements in b1, b2 and (q i ) 8 i=1 are all selected manually to guarantee that there are some expected-valueconstraintsbeingactive. 4. All the elements in R1, R2 and (q i,R ) 8 i=1 have the same distribution U( d,d) with d being a positivevalue. 5. AsforthematrixQ R ,foreachi,weletitsi-thdiagonalblocksub-matrixbea8⇥ 8zeromatrix, e.g. Q R (s i1 : s i2 ,s i1 : s i2 )= 0 8⇥ 8 , where s i1 and s i2 have already been defined before. For all otherelementsinQ R ,weleteachofthemhavethedistributionU( d,d)aswell. All the values of the random elements in this model are realized independently in our implemen- tations. Again,wehavefoldedanduploadedallthemodelconstantsonline. Thelinktoaccessthemis: http://dwz.date/cn8f. Thecorrespondingfolder’snameis"Simulationcase2: 8⇥ 8⇥ 2". Also, in case 2, it can be easily checked that the assumption (A1) holds. As for the assumption (A2), we do the similar computations like what we have done in case 1 and obtain ij for all i,j =1,2,...,N. We have uploaded the value of the matrix ˆ ⌥ with ˆ ⌥ ij = ij for all i’s and j’s online. Peoplecanaccessthematrixviathesamelink: http://dwz.date/cn8f. From the ˆ ⌥ , we can see that the assumption (A2) holds with the corresponding positive vector d 145 being (1,1,...,1) T . The model considered in case 2 is again equivalent to a deterministic one. Hence, with the same analysis in section 5.2, we can use the best-response algorithm to solve this deterministic game to obtaintheexactN.E.solutionandthenuseittogettheaccuraciesofbothalgorithms. Now, the N.E. solution of this model computed by applying the best-response algorithm is shown intable7. x ⇤ ,1 x ⇤ ,2 x ⇤ ,3 x ⇤ ,4 x ⇤ ,5 x ⇤ ,6 x ⇤ ,7 x ⇤ ,8 0.1086 0 0 0.0993 0.1472 0.1215 0.1459 0.2873 0.1905 0.0934 0.1144 0.1746 0.1701 0.2394 0.1106 0 0.1988 0.3466 0.1543 0.2531 0 0.3024 0.1219 0.0142 0.1135 0 0 0.0966 0.1204 0.3342 0.3035 0 0.0851 0.2412 0.2049 0.1421 0 0.0339 0.2051 0.2190 0.0013 0.3420 0.0589 0.0689 0.0218 0.1601 0.1277 0.3516 0.1812 0.0888 0 0.1475 0.4425 0.1938 0.0877 0.1279 0.2066 0.0042 0.2558 0.0667 0.1453 0.1340 0.1330 0.1728 Table7: N.E.solutionincase2 At the N.E. solution, the corresponding optimal dual values for each player i’s expected-value constraintsaresummarizedintable8. ⇤ ,1 ⇤ ,2 ⇤ ,3 ⇤ ,4 ⇤ ,5 ⇤ ,6 ⇤ ,7 ⇤ ,8 2.4347 0 0 0 0 0 1.4154 0 0 2.1996 0 3.9384 0 3.8258 0 0 Table8: Dualsolutionincase2 Hence there are at least 5 expected-value constraints being active at the N.E. solution over all players. Inthefollowingsub-sections,wewilllookattheperformancesofthetwoalgorithmsincase2. 146 5.3.2 Case2: LagrangianScheme Incase2,basedonthesettingsofthemodel,wehaveshownalltheassumptions(A1)–(A3)hold. Now,wecanconducttheimplementationsundertheLagrangianscheme. In case 2, all the values of ✏ O,x , ✏ O, , ✏ I,x and ✏ I, in the stopping criteria are the same with those incase1. The first thing is to tune the hyper-parameter c. To tune it, like what we have set in case 1, we let the variance of each random variable in the model be 0.004. As in the 8⇥ 8⇥ 2 model, we have much more random variables, the model actually contains more stochastic elements than the modelincase1does. Hence,toadapttothispoint,ineachouteriteration,weset ¯ L=1000. Alltheothermodelparameterskeepthesamewiththosewhenwetunedthevalueofcincase1. Incase2,wehavetheimplementationswiththevalueofcbeingselectedfromthefollowingset: {0.0001,0.0005,0.001,0.005,0.01}. Again,here,foreachvalueofc,wehavetheexperimentsfor10times. Andeachresultshownbe- lowiscomputedbytakingtheaverageofthecorresponding10results. Theinitialpointsselection ruleisthesamewithitusedincase1. Now,themainstatisticsofthetuningresultsareshownintable9. 147 c total outer iterations total inner iterations running time random samples error 0.0001 2.2 17.8 343.79 17800 0.00016766 0.0005 2 12.3 235.87 12300 0.00021329 0.001 2 11.5 222.35 11500 0.00017534 0.005 3 16.2 307.22 16200 0.00018961 0.01 3 15.8 302.25 15800 0.00024777 Table9: C’stunningincase2 Basedonthetableabove,weletcequal0.001inthefollowingsimulation. In case 2, the set of variances under which we conduct the implementations is the same with the variancessetincase1. Thus,itis: {0.0001,0.0003,0.0007,0.001,0.004,0.007,0.01,0.03,0.05}. Theincreasingstep-sizesunderdifferentvariancesaresummarizedintable10below. 148 variance Increasingstep-size ¯ L 0.0001 200 0.0003 200 0.0007 200 0.001 1000 0.004 1000 0.007 1000 0.01 10000 0.03 10000 0.05 10000 Table10: Increasingstep-sizesunderdifferentvariancesincase2 Now, given all of those settings above, the results of the simulation are shown in the following table. 149 variance total outer iterations total inner iterations running time random samples error 0.0001 2 8.8 145.38 1760 9.42E-05 0.0003 2 9.8 158.51 1960 0.00014342 0.0007 2 10.9 175.66 2180 0.00020243 0.001 2 9.3 150.84 9300 0.00011879 0.004 2 12.4 199.49 12400 0.00015713 0.007 2 13.6 219.1 13600 0.00021894 0.01 2 9.4 157.92 94000 0.00010064 0.03 2 10.5 176.69 1.05E+05 0.00019321 0.05 2 12.5 209.94 1.25E+05 0.00025493 Table11: SimulationresultsoftheLagrangianschemewithc=0.001incase2 Thedataaboveareobtainedbyrunningtheexperimentsfor10timesandthentakingtheaverages undereachvariance. Theselectionruleoftheinitialpointsisthesamewithitincase1. ThedatashownontheabovetableenhanceourconclusionsabouttheLagrangianschemeobtained in 5.2.2. And again, the above table shows that in the “8⇥ 8⇥ 2” case, the Lagrangian scheme workswellinbothtimeefficiencyandsolution’saccuracy. Now,itisthetimetochecktheperformancesofthe⇢ methodincase2. 150 5.3.3 Case2: ⇢ Method Incase2,theimplementationsinthe⇢ methodaredevelopedunderthesameschemeusedincase 1 and the values of ✏ O and ✏ I are the same with those in case 1 as well. The values of ¯ L’s under differentvariancesaresummarizedintable10. Again, in case 2, the assumption (B1) holds, which can be verified easily. From the analysis in 5.2.3, we know as long as the assumption (B2) holds under the extreme case, it will hold over all the implementations in the ⇢ method. As in case 2, the values of variances in the simulation are the same with those in case 1, we can use almost the identical logics to check the satisfication of (B2)here. The corresponding matrix ¯ 8⇥ 8 storing all the ¯ ij ’s has been uploaded online and can be accessed viathelink: http://dwz.date/cn8f. Again,whenwetaked i =1foralli’s,theassumption(B2)willholdunderthemostextremecase, whichmeans(B2)issatisfiedinalltheimplementationsincase2. Now, let’s check the performances of the ⇢ method under the 9 different model variances in this section. 151 variance total outer iterations total inner iterations running time random samples error 0.0001 3 8.4 28.238 600 9.70E-05 0.0003 3.4 9.2 30.231 680 0.00020075 0.0007 5.2 12.8 42.186 1040 0.0001873 0.001 3.8 10 33.561 3800 0.00012534 0.004 5.2 12.8 43.107 5200 0.00015349 0.007 5.9 14.2 47.85 5900 0.00028369 0.01 3.1 8.6 31.657 31000 0.00014333 0.03 5 12.4 46.135 50000 0.00014437 0.05 5.2 13 49.057 52000 0.00029657 Table12: Simulationresultsofthe⇢ methodincase2 Like what we have done in case 1, here, for each variance, we run the experiments for 10 times and take the average for each statistic to show. The initial points selection rule is the same with it usedincase1. For the ⇢ method, all the conclusions obtained in case 1 can be established again from the above table. Also,the⇢ methodoutperformstheLagrangianschemeinbothcase1andcase2. 5.4 Case3: 8⇥ 8⇥ 5 In this case, we will have a more complicated model. In this model, we still have 8 players with each playeri having 8 decision variables being represented byx i . The objective function of each 152 player will be the same with it in case 2. However, in the constraint’s side, for each player, he/she has5expected-valueconstraintsincludingsomenon-differentiableexpected-valuefunctions. The random variables appeared in each player’s constraints may have various distributions such as the uniformandthenormaldistributions. Wenamethismodelas“8⇥ 8⇥ 5”. Since in chapter 2, we have known that the Lagrangian scheme requires the differentiabilities of theexpected-valuefunctionsineachplayer’sexpected-valueconstraints,wewillonlycarryoutthe simulationforthe⇢ methodinthiscasetocultivateitspowerinsolvingamorecomplexstochastic game. 5.4.1 ModelIntroductionandGeneration Thespecificmodelformulationincase3isgivenbelow: 8 > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > : min x i 2 X i ✓ i (x i ,x i ) s.t. E[coef1(i,:)x i R1(i) b1(i)] 0 E[(coef2(i,:)+R2(i,:))x i b2(i)] 0 E[ 8 P j=1 (c ij +c R ij )·x i j 2 b3(i)] 0 E[ 8 P j=1 (e ij +e R ij )·|x i j m ij | b4(i)] 0 E[| 8 P j=1 x i j P(i)| R5(i) b5(i)] 0 9 > > > > > > > > > > > > > > > > > > > > = > > > > > > > > > > > > > > > > > > > > ; 8 i=1 . (5.4) Let’sintroducetheelementsintheabovemodelonebyone. For each i, like what we just mentioned, the objective function ✓ i (x i ,x i ) is the same with it in case2. AndtheCAP i isa8-dimensionalvectorwitheachelementbeingindependentlyrandomly generatedfromthedistributionU(0.6,0.8). 153 Now, for each player, in the first two expected-value constraints, each component in the random vector R1 has the normal distribution N(0, ) and R2 is still a 8⇥ 8 random matrix with each elementhavingthenormaldistributionN(0, ). Ineachplayer’sthirdexpected-valueconstraint,e.g. E[ 8 P j=1 (c ij +c R ij )·x i j 2 b3(i)] 0,c ij isamodel constant and c R ij is a random variable with the uniform distribution U( d,d) for each i andj. In thefourthexpected-valueconstraint,e.g. E[ 8 P j=1 (e ij +e R ij )·|x i j m ij | b4(i)] 0,bothe ij andm ij are the model constants ande R ij is a random variable with the distributionU( d,d) for eachi and j. Inthelastexpected-valueconstraintsofallplayers,e.g. E[| 8 P j=1 x i j P(i)| R5(i) b5(i)] 0, P isa8-dimensionalconstantvectorandR5isa8-dimensionalrandomvector. Alltheb1,b2,b3,b4andb5aremodelconstantshere. Let the three 8⇥ 8 matrices C, E and M store c ij , e ij and m ij for each i and j respectively and alsoletC R andE R betherandommatricestostorec R ij ande R ij foreachiandj respectively. The summarized generation methods for all the above model elements in the expected-value con- straintsareshowninthefollowingtable. 154 Element Generationmethod coef1 8⇥ 8 EachelementisgeneratedrandomlyfromthedistributionU(0.9,1.1). R1 8⇥ 1 EachcomponenthasthedistributionN(0, ). b1 8⇥ 1 Eachcomponentisselectedmanually. coef2 8⇥ 8 EachelementisgeneratedrandomlyfromthedistributionU(0.9,1.1). R2 8⇥ 8 EachelementhasthedistributionN(0, ). b2 8⇥ 1 Eachcomponentisselectedmanually. C 8⇥ 8 EachelementisgeneratedrandomlyfromthedistributionU(0.9,1.1). C R 8⇥ 8 EachelementhasthedistributionU( d,d). b3 8⇥ 1 Eachcomponentisselectedmanually. E 8⇥ 8 EachelementisgeneratedrandomlyfromthedistributionU(0.9,1.1). E R 8⇥ 8 EachelementhasthedistributionU( d,d). M 8⇥ 8 EachelementisgeneratedrandomlyfromthedistributionU(0,0.5). b4 8⇥ 1 Eachcomponentisselectedmanually. P 8⇥ 1 Eachcomponentisselectedmanually. R5 8⇥ 1 EachcomponenthasthedistributionN(0, ). b5 8⇥ 1 Eachcomponentisselectedmanually. Table13: Generationmethodsofthemodelelementsincase3 Intheabovetable,wemarkeachmodelelement’ssizeonitssubscript. We have stored and uploaded all the model constants online. They can be accessed through the link: 155 http://suo.im/61pZOM. Thenameofthefolderis“Simulationcase3: 8X8X5”. The model above is also equivalent to a deterministic one. Since both case 2 and case 3 have the sameobjectivefunctionforeachplayer,theassumption(A2)holdsinthiscase. Also,eventhough in the statement of lemma 1, we assume that the assumption (A1) holds, in the proof of lemma 1, weactuallyonlyuse(A1)toguaranteetheregionZ i , {x i 2X i |E[G i (x i ,!)] 0}isnon-empty, closed and convex for alli’s. Here, in case 3, for each playeri, we can give one Slater’s point as theexampletoshowthatZ i isnon-empty. The following table shows one feasible point for each i. Using the model constant elements ac- cessed in the above link, we can guarantee that the following points are indeed the Slater’s points definedat(A1.6)or(B1.6). ˆ x 1 ˆ x 2 ˆ x 3 ˆ x 4 ˆ x 5 ˆ x 6 ˆ x 7 ˆ x 8 0.1356 0.0780 0.1516 0.1051 0.0648 0.2245 0.0909 0.1811 0.1517 0.1679 0.1202 0.1181 0.1647 0.1020 0.1396 0.1025 0.1553 0.1543 0.1952 0.1213 0.1317 0.1815 0.2422 0.0729 0.1504 0.1331 0.0530 0.1473 0.1170 0.1697 0.0901 0.1158 0.1424 0.1053 0.1768 0.1468 0.1065 0.1596 0.1416 0.1590 0.2026 0.1171 0.1855 0.0742 0.1590 0.1167 0.0946 0.0723 0.2074 0.0528 0.1683 0.0865 0.1537 0.1781 0.1848 0.2234 0.1852 0.0846 0.1854 0.1038 0.1849 0.1395 0.0992 0.1263 Table14: OnesetofSlater’spointsofallplayersincase3 156 Also,foreachi,theconvexityandclosenessofZ i areeasytobeverified. Hence, based on lemma 1, we can apply the best-response algorithm to obtain the unique N.E. solutionoftheequivalentdeterministicgameincase3. ThetablebelowshowstheN.E.solutionofthisdeterministicgame. x ⇤ ,1 x ⇤ ,2 x ⇤ ,3 x ⇤ ,4 x ⇤ ,5 x ⇤ ,6 x ⇤ ,7 x ⇤ ,8 0.1369 0.0062 0 0.10812 0.1455 0.1039 0.1254 0.3048 0.2173 0.1073 0.1604 0.1764 0.1679 0.2221 0.1063 0 0.2242 0.3342 0.1959 0.2442 0 0.2846 0.1293 0.0332 0.1386 0.0016 0 0.1025 0.1183 0.3080 0.2757 0 0.1100 0.2337 0.2464 0.1360 0 0.0219 0.1793 0.2280 0.0242 0.3290 0.1066 0.0610 0.0199 0.1558 0.1022 0.3280 0.2094 0.1000 0 0.1588 0.4406 0.1833 0.0946 0.1453 0.2363 0.0206 0.3008 0.0681 0.1437 0.1078 0.1115 0.1660 Table15: N.E.solutionincase3 Thecorrespondingoptimaldualsolutionsareshownbelow. 157 ⇤ ,1 ⇤ ,2 ⇤ ,3 ⇤ ,4 ⇤ ,5 ⇤ ,6 ⇤ ,7 ⇤ ,8 0 0 0 0 0 5.7744 1.979 0 0 0 0 4.0851 0 0 0 0 0 5.2076 0 0 0 0 0 0 0 0 6.1213 0 0 0 2.0451 2.4417 1.5028 0 0 0 0 0 0 0 Table16: Dualsolutionincase3 Inthetableabove,wecanseethattheadditionsofmoreexpected-valueconstraintsineachplayer’s sub-problemaremeaningful. 5.4.2 Case3: ⇢ Method In case 3, we will investigate the ⇢ method’s performance under each model variance. The set of used model variances is the same with it in case 2 and the values of ✏ O and ✏ I are the same with those in cases 1 and 2 as well. Also, the increasing step-size ¯ L remains the same under each variance. Before we demonstrate the performances of the ⇢ method in this section, let’s talk about the as- sumptions(B1)and(B2)atfirst. Since in case 3, the objective function remains the same with it in case 2 for each playeri, the as- sumption (B2) holds provided that we carry out the implementations under the same set of model variances. Asfortheassumption(B1),itisroutinetoverifytheassumptions(B1.1)to(B1.5)hold. Andin5.4.1,wehaveshowntheassumption(B1.6)holdsaswell. 158 Thus, both assumptions (B1) and (B2)hold in case 3, which means we can have thesimulation of the⇢ methodwiththetheoreticallyguaranteedconvergenceproperties. Again, in this case, for each variance, we run the experiments for 10 times and then take the aver- age for each statistic to show. For each variance, the set of 10 initial points in the corresponding 10experimentskeepsthesame. Now,let’slookattheperformancesofthe⇢ methodincase3. variance total outer iterations total inner iterations running time random samples error 0.0001 3.4 8.8 55.025 680 0.0001245 0.0003 4.4 10.8 66.649 880 0.00017113 0.0007 5.2 12.4 76.288 1040 0.00027139 0.001 4 10 62.238 4000 0.0001336 0.004 6.7 15.4 94.843 6700 0.0002278 0.007 7.9 17.8 109.3 7900 0.00026882 0.01 4.2 10.4 69.05 42000 0.00013808 0.03 4.6 11.2 74.437 46000 0.00025781 0.05 6 14 94.723 60000 0.00020741 Table17: Simulationresultsofthe⇢ methodincase3 Again,fromthetableabove,wecanseethatthe⇢ methodachievesverygoodsolutions’accuracies andonlyneedstherelativelysmallnumbersofi.i.d. randomsamplesunderallvariances. 159 5.5 Conclusions Basedonalltheimplementationsfromcase1tocase3,wecanseethatthe⇢ methodworksbetter thantheLagrangianscheme. OnethingtohighlightisthattheCVXpackageusedinoursimulation cannotdealwiththecasewheretheobjectivefunctionislikex 2 x 2 2 thatisdefinitelyconvex. The CVX package we are using here requires all the independent terms in the objective function to be convex. Nevertheless, in our simulation of the Lagrangian scheme, in each inner iteration, for each player i, due to the existence of the term c 2 k i ˜ v 1,i k 2 , when we substitute i j by µ+1,v,i j (x i ), argmax j 0 j Lµ · Lµ P s=1 G i j (x i ,! s ) c 2 ( j ˜ v 1,i j ) 2 =max(0, 1 c·Lµ Lµ P s=1 G i j (x i ,! s )+ ˜ v 1 j ) for j =1,2,3,...,k i , some non-convex terms will appear on the player i’s objective function even though the whole function is guaranteed to be convex. As a result, the CVX cannot make computations automatically. To address this problem, for each expected-value constraint in the SAA reformulation, we have to split each optimization problem to two sub-problems such that each one’s objective function does not contain any non-convex term. However, the expense of doing that is the increase of the simulation time. Now, in both case 1 and case 2, for each player i, we have two expected-value constraints, which means for each optimization problem, we have to split it to four separate ones and thus make the simulation time be almost the 4 times of the simulation time when the splittings are not needed at all. Therefore, to make the comparisons between the ⇢ method and the Lagrangian scheme more fair, we should divide each running time oftheLagrangianschemeby4. However, based on the performances in case 1 and case 2, even though we divide each running time under the Lagrangian scheme by 4, we can still see that the overall time efficiency of the Lagrangianschemeisnotbetterthantheoveralltimeefficiencyofthe⇢ method. 160 Actually,inthesimulationoftheLagrangianscheme,ineachinneriteration,tosolveeachplayer’s optimization problem, we can employ the analysis of the Linear Complementary Problem (LCP) [8,12] to develop our implementations, which has the great potential to avoid the above issue. However,forconvenience,wedonotcarryoutourimplementationsusingLCPanalysishere,and hence the method we are using to solve each player’s optimization problem in the Lagrangian scheme is not very time-efficient. Therefore, when we look at the simulation results in different algorithms, it is more reasonable to focus on two metrics. The first one is the number of the totalinneriterationsandthesecondoneisthenumberofthetotalgeneratedi.i.d. randomsamples. Thesetwomethodshavethesimilarperformancesinthefirstmetric,whilethe⇢ methoddominates theLagrangianschemeinthelightofthesecondmetric. The main reason for us to introduce the Lagrangian scheme is that the assumption (B2) in the ⇢ method is very strong. If in some players’ objective functions, the value ranges of some random variables are very large, the assumption (B2) will bemuch likely violated, whichmay make some inner-loops not converge with the positive probability. However, in the Lagrangian scheme, we don’t need to worry about this issue, because each inner-loop in the Lagrangian scheme actually solves a max-min game with the expected-value objective functions, which means as long as the assumptions(A1)and(A2)hold,theinner-loopsolutionssequencemustconvergetothedesirable pointina.e. regardlessofthevaluerangesoftherandomvariablesinallplayers’objectives. Hence, when all the model’s expected-value constraints are differentiable and the value ranges of the random variables in the objectives are large enough so that the assumption (B2) could be violated with a very high confidence, we recommend to use the Lagrangian scheme provided that all other assumptions are satisfied. Otherwise, we recommend to use the ⇢ method, especially whensomenon-differentiablerandomfunctionsareincludedintheexpected-valueconstraints. 161 6 FutureWork In this thesis, we develop two algorithms to solve the stochastic games with expected-value con- straints and conduct the simulations in different cases for those two methods. It turns out that bothalgorithmshavethetheoreticalconvergencepropertiesandthegoodsimulationperformances. However,therearestillsomeroomfortheimprovementsinbothmethods. 1. Recalling that in the Lagrangian scheme, we need to assume the differentiabilities of the expected-value functions in each player’s expected-value constraints, e.g. E[G i (x i ,!)]. Actu- ally,thereisagreatpotentialtoremovethisassumption. The main difficulty of this removal is that in the outer-loop analysis including both convergence and rate proofs, our deductions mainly rely on the analysis of the V.I. problem, e.g. VI( ˜ K, ˜ F), where requires ˜ F to be a single-valued mapping, instead of a set-valued mapping based on [12]. However, when E[G i (x i ,!)] is non-differentiable for some i, we need to use the sub-gradients of E[G i (x i ,!)] to replacer x iE[G i (x i ,!)] in our analysis. Thus, ˜ F cannot be guaranteed to be single-valued. Hence, to obtain the similar results, we need to re-construct the V.I. analysis where ˜ F isnotnecessarytobeasingle-valuedmapping. In the future work, we hope that we can extend the V.I. analysis correspondingly and allow each player’sexpected-valuefunctionsinhis/herexpected-valueconstraintstobenon-differentiable. 2. In chapter 5, we have shown that the ⇢ method has the better performance than the Lagrangian scheme, provided that all the assumptions in both algorithms hold. However, on the theoretical side, it is tough to accomplish the rate analysis of the ⇢ method. The main reason is that in the ⇢ 162 method, for each playeri, we add one dummy variable ⇢ i and penalize it via adding it on his/her objectivefunctionwiththecoefficientbeingL v . Hence,underaspecificouteriterationv,foreach playeri, his/her optimal strategy, e.g. (x 1 ,v,i ,⇢ 1 ,v,i ) with ⇢ 1 ,v,i =max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x 1 ,v,i ,! s ), depends on both the others’ optimal strategies, e.g. x 1 ,v, i , and the relative relation between the function 1 Lv Lv P s=1 H i (x i ,x i ,! s )andthefunctionL v · max j=1,2,...,k i 1 Lv Lv P s=1 G i j (x i ,! s ). Therefore,therate analysis becomes more complicated. What’s more, in the Lagrangian scheme, we can use the mature V.I. analysis to establish the outer-loop rate analysis. While, in the ⇢ method, the feasible region of each player’s sub-problem varies as the outer-loop iterates, which means we cannot use the same trick that is used in the Lagrangian scheme, to employ the V.I. analysis to help us estab- lishtherateanalysisofthe⇢ method. Aspartofthefuturework,wehopewecanaddressthosedifficultiesmentionedabovetosuccess- fullyextendthetheoreticalresultsinthe⇢ method,e.g. havingtherateanalysis. 163 References [1] J.Atlason,M.EpelmanandG.Henderson.Callcenterstangwithsimulationandcuttingplane methods.AnnalsofOperationsResearch127,333-358(2004). [2] A. Bensoussan. Points de Nash dans le cas de fontionnelles quadratiques et jeux dierentiels lineairesaNpersonnes.SIAMJournalonControl12,460-499(1974). [3] S. Berridge and J. Krawczyk. Relaxation algorithms in finding Nash equilibria. Economic WorkingPapersArchive,(1997). [4] D.P. Bertsekas, A. Nedic and A.E. Ozdaglar. Convex Analysis And Optimization. Athena Scientific,(2003). [5] J.R. Birge and F. Louveaux. Introduction to stochastic programming. Springer Series in Op- erationsResearch(Springer,NewYork),(1997). [6] S.BoydandL.Vandenberghe.ConvexOptimization. CambridgeUniversityPress,(2004). [7] E. Cavazzut, M. Pappalardo and M. Passacantando. Nash equilibria, variational inequali- ties,anddynamicalsystems. Journal of Optimization Theory and Applications114,491-506 (2002). [8] R.W.Cottle,J.S.PangandR.E.Stone.Thelinearcomplementarityproblem.AcademicPress, (1992). [9] G.B. Dantzig. Linear programming under uncertainty. Management Science 1, 197-206 (1955). 164 [10] F.Facchinei,A.FischerandV.Piccialli.OngeneralizedNashgamesandvariationalinequal- ities.OperationsResearchLetters35,159-164(2007). [11] F. Facchinei and C. Kanzow. Generalized Nash equilibrium problems. Annals of Operations Research175,177-211(2010). [12] F.FacchineiandJ.S.Pang.Finite-DimensionalVariationalInequalitiesandComplementarity Problems.(Springer,NewYork),(2003). [13] F. Facchinei and J.S. Pang. Nash equilibria: The variational approach. In D. Palomar and Y. Eldar, editors. Convex Optimization in Signal Processing and Communications. Cambridge UniversityPress(Cambridge,England),443-493(2009). [14] G.B. Folland. Real Analysis: Modern Techniques and Their Applications, Second Edition. NewYork: AWiley-IntersciencePublication,(1999). [15] JoelN.Franklin.MatrixTheory.DoverPublications,(1993). [16] M.Fukushima.Aclassofgapfunctionsforquasi-variationalinequalityproblems.Journalof IndustrialandManagementOptimization3,165-171(2007). [17] G. Gurkan, O. Ozdemir and Y. Smeers. Generation capacity investments in electricity mar- kets: Perfectcompetition(CentERDiscussionPaper;Vol.2013-045).Tilburg: Econometrics, (2013). [18] P.T.Harker.GeneralizedNashgamesandquasi-variationalinequalities.EuropeanJournalof OperationalResearch54,81-94(1991). 165 [19] P.T. Harker and J.S. Pang. Finite-dimensional variational inequality and nonlinear comple- mentarity problems: A survey of theory, algorithms and applications. Mathematical Pro- grammingSeriesB48,161-220(1990). [20] J.HigleandS.Sen.StochasticDecomposition: AStatisticalMethodforLargeScaleStochas- ticLinearProgramming.(Springer,NewYork),(1996). [21] H. Jiang and H. Xu. Stochastic approximation approaches to the stochastic variational in- equalityproblem.IEEETransactionsonAutomaticControl53,1462-1475(2008). [22] P. Kall and J. Mayer. Stochastic Linear Programming. International Series in Operations ResearchandManagementScience156 (Springer,NewYork),(2011). [23] P.KallandS.W.Wallace.StochasticProgramming.JohnWileyandSons,Chichester(1994). [24] J.B. Krawczyk. Coupled constraint Nash equilibria in environmental games. Resource and EnergyEconomics,vol.27,157-81(2005). [25] J.B. Krawczyk and S. Uryasev. Relaxation algorithm to find Nash equilibria with economic applications.EnvironmentalModelingandAssessment,vol.5,63-73(2000). [26] D. Kuhn. Convergent bounds for stochastic programs with expected value constraints. The StochasticProgrammingE-PrintSeries(SPEPS),(2007). [27] J.Lei,U.V.Shanbhag,J.S.PangandS.Sen.Onsynchronous,asynchronous,andrandomized best response schemes for computing equilibria in stochastic Nash games. Mathematics of OperationsResearch,(2018). 166 [28] J. Liu, Y. Cui, J.S. Pang and S. Sen. Two-stages stochastic programming with linearly bi- parameterizedquadraticresource.Preprint,(2018). [29] J. Liu and S. Sen. Asymptotic results of stochastic decomposition for two-stage stochastic quadraticprogramming.SIAMJournalonOptimization,(accepted)(2019). [30] T.L.MagnantiandG.Perakis.Averagingschemesforvariationalinequalitiesandsystemsof equations.MathematicsofOperationsResearch,vol.22,568-587(1997). [31] T.L. Magnanti and G. Perakis. Computing fixed points by averaging in Transportation and NetworkAnalysis-CurrentTrends.P.MarcotteandM.Gendreau,eds.(Springer,NewYork), (2001). [32] J.F. Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences36,48-49(1950). [33] J.F.Nash.Non-cooperativegames. AnnalsofMathematics54,286-295(1951). [34] M. O’Brien. Techniques for Incorporating Expected Value Constraints into Stochastic Pro- grams.PhDThesis,StandardUniversity,(2000). [35] J.S. Pang, S. Sen and U.V. Shanbhag. Two-stage non-cooperative games with risk-averse players.MathematicalProgramming165(1),235-290(2017). [36] U. Ravat and U.V. Shanbhag. On the characterization of solution sets of smooth and non- smooth convex stochastic Nash games. SIAM Journal on Optimization 21 (3), 1168-1199 (2011). 167 [37] R.T.RockafellarandJ.Sun.Solvingmonotonestochasticvariationalinequalityandstochas- tic complementarity problems by progressive hedging. Mathematical Programming 174 (1), 453-471(2019). [38] R.T.RockafellarandR.J-B.Wets.Variationalanalysis.Berlin: Springer,(1998). [39] A.RuszczynskiandA.Shapiro.Introduction.InHandbookinOperationsResearchandMan- agementScience,vol.10,ElsevierScience(Amsterdam),1-64(2003). [40] A. Shapiro, D. Dentcheva and A. Ruszczynski. Lectures on Stochastic Programming: Mod- elingandTheory.SIAMPublications,Philadelphia,(2009). [41] VincentY.F.Tan.SimplifiedproofofSlater’stheoremforstrongduality.E-printavailableat: https://www.ece.nus.edu.sg/stfpage/vtan/ee5138/slater.pdf,(2015). [42] S.UryasevandR.Y.Rubinstein.Onrelaxationalgorithmsincomputationofnon-cooperative equilibria.IEEETransactionson AutomaticControl,vol.39,1263-1267(1994). [43] W. Wang. Sample average approximation of risk-averse stochastic programs. PhD thesis, GeorgiaInstituteofTechnology,(2007). [44] W. Wang and S. Ahmed. Sample average approximation of expected value constrained stochasticprograms.Operations ResearchLetters36,515-519(2008). [45] H.Xu.Sampleaverageapproximationmethodsforaclassofstochasticvariationalinequality problems.Asia-Pac.J.Oper.Res.27,103-119(2010). [46] H.XuandD.Zhang.StochasticNashEquilibriumproblems: sampleaverageapproximation andapplications.ComputationalOptimizationandApplications,55Issue3,597-645(2013). 168

Abstract (if available)

Abstract Stochastic game is a combination of stochastic programming and game theory. We propose two methods solving the stochastic games with expected-value constraints. The first one is called the "Lagrangian Scheme". The second one is called the "ρ method". In the Lagrangian scheme, for each player, we relax his/her expected-value constraints by augmenting the objective with a weighted sum of the expected-value constraint functions and the weights are called the "Dual Variables". After this modification, the game becomes a max-min game which can be proved to be equivalent to a "Variational Inequality" (V.I.) problem. With the monotonicity assumption on the equivalent V.I. problem, we can solve it using the "Proximal Point" (P.P.) method whose whole iterations form the outer-loop of the Lagrangian scheme. In each outer iteration, we develop an inner-loop to solve a new max-min game by combining the "Best-Response" (BR) algorithm and the "Sample Average Approximation" (SAA) method. Therefore, the inner-loop solutions sequence is a sequence of the solutions of sampled max-min problems. In the ρ method, we have two loops as well. In each outer iteration, we use the SAA method to relax all the expected-value functions and then add one extra dummy variable in each player’s sub-problem to avoid the infeasibility problem incurred by the relaxations. The sample size used in the SAA method increases as the outer-loop iterates. To solve the formed deterministic game in each outer iteration, we apply the BR algorithm directly under some assumptions on the BR mapping. Our contributions include the analyses of the convergence properties of both the Lagrangian scheme and the ρ method. Also, we provide the convergence rate analysis of the Lagrangian method. Computational results are presented as well to illustrate the validity and efficiency of each algorithm. Furthermore, some interesting future works that can extend the current theoretical results are proposed.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Topics in algorithms for new classes of non-cooperative games

PDF

On the interplay between stochastic programming, non-parametric statistics, and nonconvex optimization

PDF

Algorithms and landscape analysis for generative and adversarial learning

PDF

Landscape analysis and algorithms for large scale non-convex optimization

PDF

Mixed-integer nonlinear programming with binary variables

PDF

Difference-of-convex learning: optimization with non-convex sparsity functions

PDF

Computational stochastic programming with stochastic decomposition

PDF

Provable reinforcement learning for constrained and multi-agent control systems

Asset Metadata

Creator Zhou, Yunan (author)

Core Title Stochastic games with expected-value constraints

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Industrial and Systems Engineering

Publication Date 11/29/2020

Defense Date 08/14/2020

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag almost everywhere,best-response algorithm,diagonal dominance,expected-value constraints,Lagrangian function,Nash equilibrium,OAI-PMH Harvest,sample average approximation,stochastic games,variational inequality problem

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Pang, Jong-Shi (committee chair), Carlsson, John Gunnar (committee member), Jain, Rahul (committee member), Razaviyayn, Meisam (committee member), Sen, Suvrajeet (committee member)

Creator Email yunanzho@usc.edu,zhouyunan0428@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-400311

Unique identifier UC11668649

Identifier etd-ZhouYunan-9164.pdf (filename),usctheses-c89-400311 (legacy record id)

Legacy Identifier etd-ZhouYunan-9164.pdf

Dmrecord 400311

Document Type Dissertation

Rights Zhou, Yunan

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA