Close
About
FAQ
Home
Collections
Login
USC Login
Register
0
Selected
Invert selection
Deselect all
Deselect all
Click here to refresh results
Click here to refresh results
USC
/
Digital Library
/
University of Southern California Dissertations and Theses
/
Sequential testing of multiple hypotheses with FDR control
(USC Thesis Other)
Sequential testing of multiple hypotheses with FDR control
PDF
Download
Share
Open document
Flip pages
Contact Us
Contact Us
Copy asset link
Request this asset
Transcript (if available)
Content
Sequential T esting of Multiple Hypotheses With FDR Control Michael Hankin Ad visor: Ja y Bar tr off; Committee: Jinchi L v and Gar y R osen Ph. D. in Applied Ma thema tics, University of Southern Calif ornia A ugust 17, 2017 Contents 0.1. A c kno wlegemen ts 5 Chapter 1. Sequen tial Step-do wn T esting of Multiple Hyp otheses Under Arbitrary Join t Distributions: An Optimization Approac h to FDR Con trol 6 1.1. Motiv ation 6 1.2. In tro duction and Prior W ork 7 1.3. Main Results 14 1.4. Sim ulation Studies and Data Analysis 39 1.5. Conclusions 54 Chapter 2. FDR Con trol With P ositiv e Dep endence 55 2.1. In tro duction and Motiv ation 55 2.2. Sequen tial Step-up Pro cedures 56 2.3. FDR Con trol Under P ositiv e Dep endency 61 2.4. Sim ulations 66 2.5. Conclusions and Discussion 73 Chapter 3. MultSeq Python P ac k age 75 3.1. In tro duction 75 3.2. Discussion 82 Bibliograph y 83 App endix A. App endix for Chapter 1 86 A.1. Cuto Estimation and Notes 86 A.2. Explicit Stopping Times 92 A.3. BH Cuto Example 93 App endix B. App endix for Chapter 2 94 B.1. PRDS Equalit y-Inequalit y Lemma 94 3 CONTENTS 4 App endix C. App endix for Chapter 3 97 C.1. Roadmap 97 0.1. A CKNO WLEGEMENTS 5 0.1. A c kno wlegemen ts First and foremost, I w ould lik e to thank m y advisor, Ja y Bartro, for taking me on as a studen t so late in the game and for exp ertly guiding me from start to nish. Also deserving of m y gratitude are the other facult y I’v e w ork ed with and learned so m uc h from. Gary Rosen who, along his colleague Ch unming W ang, help ed shepherd me through m y rst few y ears of graduate sc ho ol, and has acted as a men tor and friend; Y an Liu, who help ed to exp ose me to a wide arra y of topics and to ols that ha v e b een instrumen tal in furthering m y career; she w as also instrumen tal in teac hing to read, write, and appreciate academic researc h; and Jinc hi Lv who exp osed me to the deeply mathematical side of high dimensional statistics, helping me to recognize m y passion for the eld. I w ould also lik e to thank Larry Goldstein, Ja y Batro, Jinc hi Lv, Y an Liu, and F ei Sha for bringing to life the courses they taugh t in suc h a w a y as to sw a y me from loathing statistics to ha ving trouble imagining doing an ything other than statistics for the rest of m y life life. The excitemen t ab out and deep understanding of the topic they pro vided me with ha v e c hanged the course of m y life, and set me on a career tra jectory that I an ticipate b eing incredibly fullling, if the rst leg of it is an y indication. F urther, I w ould lik e to thank the USC Mathematics Departmen t, in particular Am y Y ung and Arnold Deal, for their help and encouragemen t o v er the y ears. I am also deeply appreciativ e of the departmen t for sp onsoring m y tra v el and fees to allo w me to presen t the researc h in this thesis to the Multiple Con trol Pro cedures Conference (MCP 2017) at UC Riv erside, via the USC Mathematics Departmen t Graduate Dev elopmen t A w ard, and of the National Science F oundation and Ja y Bartro for purc hasing the computer necessary to run the sim ulations in this thesis via National Science F oundation gran t DMS-1310127. I could not ha v e made it through m y PhD without the lo v e, supp ort, encouragemen t, and distractions pro vided b y m y friends, family , and dogs. I can’t list them all here, but I’d lik e to giv e sp ecial thanks to m y mother, Jo di Cohn, m y father, Marc Hankin, m y dog Mogli, and m y friend Danielle V alner. These four, in particular, w ere instrumen tal in k eeping me fo cused and getting me to this p oin t. Finally , I w ould lik e to thank m y committee mem b ers again, for b eing in v olv ed in this nal leg of m y graduate studies. All three mem b ers, Ja y Bartro, Jinc hi Lv, and Gary Rosen, had con tributed signican tly to m y progress long b efore I b egan w orking on m y actual dissertation, and I am deligh ted to ha v e them b e there to see it nally come together. CHAPTER 1 Sequential Step-down T esting of Multiple Hypotheses Under Arbitrary Joint Distributions: An Optimization Approach to FDR Control 1.1. Motiv ation As the v olume of data generated daily skyro c k ets, our metho ds of analysis m ust also mature in suc h a manner as to most ecien tly dra w useful insigh ts from the man y streams of information a v ailable to us. The sim ultaneous testing of m ultiple h yp otheses and the use of streaming data for h yp othesis testing are t w o suc h highly relev an t, w ell studied areas. Ho w ev er, their in tersection has only b egun to b e explored, and presen ts a m ultitude of opp ortunities to further exploit the data a v ailable to us. Sequen tial statistical metho ds are pro cedures under whic h data is collected progressiv ely , generally stop- ping once some statistic reac hes a pre-sp ecied signicance lev el. This allo ws for m uc h more ecien t ex- p erimen ts, as a xed-sample statistical pro cedure ma y under- or o v er-collect, resulting in an underp o w ered test or unnecessary o v er-exp enditures, resp ectiv ely . Multiple testing addresses the issue of sim ultaneously testing m ultiple h yp otheses while con trolling their t yp e 1 error, whic h migh t app ear to b e a trivial problem. Bennett et al. (2009) pro v e otherwise b y nding neural activit y in the brain of a dead sh as a result of failing to correct for m ultiple testing errors when sim ultaneously lo oking for activit y in a large n um b er of v o xels. Naiv e solutions, on the other hand, ma y lea v e the practitioner with a test so conserv ativ e or underp o w ered as to b e essen tially useless as describ ed in Benjamini and Ho c h b erg (1995) and demonstrated in the example regarding the Bonferroni correction in the follo wing section. These disciplines ha v e b een com bined in a v ariet y of no v el manners for m yriad applications, including m ulti-endp oin t clinical studies wherein a n um b er of treatmen ts are b eing ev aluated on a gro wing group of sub jects (or on a xed group of sub jects o v er time). Data collection on the individual treatmen ts ma y b e stopp ed once they ha v e b een declared eectiv e or ineectiv e, allo wing the resources of the study to b e applied to the remaining undecided h yp otheses. In this c hapter, w e will presen t w ork that guaran tees a b ound on a generalization of t yp e 1 error to sequen tial testing of m ultiple h yp otheses without an y assumptions ab out the join t distribution of the test statistics. This error metric, called the F alse Disco v ery Rate, (FDR), in tro duced in Benjamini and 6 1.2. INTR ODUCTION AND PRIOR W ORK 7 Ho c h b erg (1995), addresses the p ortion of disco v eries that w e should consider to b e truly signican t. W e formally dene the metric in Section 1.2 and demonstrate its con trol in Section 1.3 b y viewing the FDR of a statistical testing pro cedure through the lens of optimization, maximizing it under standard conditions on the marginal distributions of the test statistics and a sp ecied statistical pro cedure, without an y requiremen ts regarding the join t distribution. Finally , in Section 1.4 w e apply our framew ork to a v ariet y of sim ulations of sequen tial data, including sim ulations based on a real database of drug side eect rep orts where our pro cedure is used to detect drugs that ma y cause amnesia. 1.2. In tro duction and Prior W ork One of the main to ols a v ailable to a statistician is the h yp othesis test. Casella and Berger (2002, p 373, Denition 8.1.1) denes a h yp othesis to b e a statemen t ab out a p opulation parameter, and a h yp othesis test to b e a rule that sp ecies for whic h samples a n ull h yp othesis H 0 is to b e accepted, and for whic h v alues it is to b e rejected; the probabilit y of obtaining a sample that indicates rejection is called the T yp e 1 Error of a test. The statistician observ es the data and uses it to dra w some conclusion ab out the v eracit y of the n ull h yp othesis, with a degree of condence tied to the t yp e 1 error rate, but is generally unable to directly observ e the true v alue of the p opulation parameter, up on whic h the h yp othesis dep ends. 1.2.0.1. Multiple T esting. In m ultiple testing, the rst h urdle is to determine the manner in whic h t yp e 1 error is generalized. The classical and most ob vious approac h is to limit the probabilit y that a single true n ull h yp othesis is incorrectly rejected, referred to as F amily Wise Error Rate. The most classical approac h to con trolling this error metric, hereinafter referred to as FWER, is kno wn as the Bonferroni (1936) correction. If w e are testing m h yp otheses of whic h an unkno wn n um b er m 0 are true and the remainder m 1 (where m 1 =mm 0 ) are false, w e ma y wish to con trol FWER at lev el : T o do so w e ma y tak e an y ( i ) m i=1 suc h that i 0 and P m i=1 i = , although the standard Bonferroni approac h uses i = =m for all i. F or simplicit y w e assume that the rst m 0 h yp otheses are the true n ulls. W e also assume v alid p-v alues (p i ) m i=1 for eac h h yp othesis, where if the i th n ull is true then P (p i a) a for all a2 (0; 1) (Casella and Berger, 2002). Using Bo ole’s inequalit y , w e obtain the follo wing b ound: FWER =P ([ m0 i=1 fp i i g) m0 X i=1 P (p i i ) m0 X i=1 i m X i=1 i =: While this correction can b e applied without an y assumptions on the join t distribution of the test statistics, it greatly reduces the p o w er of the tests. F or this reason, more n uanced approac hes w ere required. Holm (1979) and Ho c h b erg (1988) presen ted strong initial attempts to con trol FWER b y rejecting the k n ull h yp otheses with the most signican t statistics and accepting the others where k is a function of the ordered 1.2. INTR ODUCTION AND PRIOR W ORK 8 set of p-v alues. In these sc hemes k Hochberg = max i2f1;:::;mg :p (i) i=m ; tak en to b e 0 when that set is empt y , and k Holm = min i :p (i) >=(mi + 1) 1; tak en to b e m when the set is empt y , where p (i) m i=1 corresp ond to the v alue-ordered p-v alues, p (1) p (2) :::p (m) . These t yp es of m ultiple rejection criteria, whic h w e dene next, are examples of step-up (SU) and step-do wn (SD) pro cedures resp ectiv ely . Fixed-sample step-up and step-do wn pro cedures tak e as input an increasing set of cutos, thresholds b et w een 0 and 1 (1.2.1) 0 = 0 < 1 ::: m < 1 (with 0 sp ecied for notational con v enience) referred to as ~ := ( i ) m i=1 , and the p-v alues. They return a rejection threshold b elo w whic h all p-v alues imply rejection of their n ull h yp otheses or, equiv alen tly , the set of rejected n ull h yp otheses. Both t yp es of pro cedures uses the ordered list of p-v alues to c ho ose their threshold, but do so using dieren t criteria, resulting in dieren t error con trol and p o w er prop erties. Step- up pro cedures c ho ose that threshold b y stepping up from the least signican t result, failing to reject n ull h yp otheses un til they reac h the least signican t n ull for whic h the rejection criteria is satised, rejecting it and an y more signican t results. This c hapter will fo cus on step-do wn pro cedures, whic h step do wn from the most signican t h yp othesis, rejecting eac h un til it reac hes the rst h yp othesis that fails to trip its cuto. That is to sa y w e b egin with i = 1 and reject if p (i) < i ; w e then incremen t i and reject the corresp onding h yp otheses if they satisfy the condition un til one fails to do so. See Figure 1.2.1 for a comparison of xed- sample size, rejectiv e step-up and step-do wn pro cedures; b y rejectiv e w e mean that the rejection criteria are pro vided explicitly , and an y n ull h yp otheses not rejected are assumed to b e accepted. Here, b ecause the least signican t p-v alue is more signican t than the nal cuto, it is rejected along with all the more signican t h yp otheses under step-up. Under step-do wn, only the t w o most signican t h yp otheses are rejected, b ecause the third most signican t p-v alue fails to trip its cuto. W e ma y then dene the n um b er of rejections under general xed-sample step-up and step-do wn pro cedures as follo ws: 1.2. INTR ODUCTION AND PRIOR W ORK 9 (1.2.2) k SU = max i2f1;:::;mg :p (i) i (1.2.3) k SD = min i2f1;:::;mg :p (i) > i 1; with the same empt y set corrections as k Holm and k Hochberg , ab o v e. These t yp es of pro cedures are most easily in tro duced using the p-v alue framew ork, but are just as applicable to more general test statistic framew orks. If w e tak e i to b e the test statistic for the i th n ull h yp othesis, with high v alues indicating rejection, then w e require m 2 v aluesfA i;j g m i;j=1 suc h that if the i th n ull h yp othesis is true then P i A i;j j for i;j2f1;:::;mg. W e will generalize this framew ork to the sequen tial setting, though that can easily b e con v erted bac k to p-v alues. F or simplicit y , w e assume a degree of standardization across the test statistics (akin to that pro vided b y the p-v alue approac h) suc h that for all j2f1;:::;mg A j :=A 1;j =A 2;j =::: =A m;j . In the case of the class of tests referred to as SPR T s this is gran ted via the W ald appro ximations; b oth concepts are in tro duced b elo w. Figure 1.2.1. Fixed-sample step-up vs. step-do wn example and comparison. The hori- zon tal lines are the cutos for the p-v alues, the p oin ts are the actual p-v alues, and the annotations explain whether step-up (SU) and step-do wn (SD) w ould reject that p oin t. The increase in genome-wide asso ciation studies prompted the need for a more exible extension of t yp e 1 error con trols to m ultiple h yp othesis testing framew orks (see Bartro and Song, 2013). One suc h metric is the aforemen tioned FDR, the exp ected v alue of the ratio of the n um b er of true n ulls rejectedreferred to as V , to the total n um b er of n ulls rejected, referred to as R. F or an y m ultiple testing pro cedure, b e it xed 1.2. INTR ODUCTION AND PRIOR W ORK 10 or sequen tial, that reac hes an acceptance or rejection decision for eac h of the m n ull h yp otheses, Benjamini and Ho c h b erg (1995) dene the F alse Disco v ery Prop ortion (FDP) as the random ratio of false rejections to total rejections, and FDR as its exp ectation: (1.2.4) FDP :=V= (R_ 1) (1.2.5) FDR :=E[V=(R_ 1)] =E [FDP] where_ is the maxim um op erator and ensures that the FDP is alw a ys dened b y setting it to 0 when there are no rejections. The random v alue FDP is a function of b oth the realized data and the pro cedure itself in that V and R are random realizations of the n um b er of h yp otheses rejected b y the pro cedure in question, whose distribution dep ends on that of the underlying data; FDR is a deterministic function of the data distribution and the pro cedure. Con trolling FDR when there are a large n um b er of h yp otheses b eing tested allo ws the user to main tain faith in the results of the pro cedure, in the sense that the rate of false disco v eries is con trolled on a v erage, while greatly increasing the p o w er o v er that of FWER con trolled pro cedures. FWER con trolling pro cedures consider all decisions in v olving an y false rejections to b e equally acrimonious; Bennett et al. (2009) discuss ho w, in high dimensional settings, they ma y o v erly p enalize a largely accurate rejection decision. A dditionally , FWER measures ho w lik ely a false rejection is to o ccur giv en that a n ull h yp othesis is true, something ab out whic h the user cannot kno w. FDR pro cedures w eigh the sev erit y of imp erfect decisions in a manner that lends itself to directly determining ho w m uc h to trust a rejection, something that is kno wn b y the user. W e use trust in to mean that if the user w ere to randomly dra w a h yp othesis from the set of rejected h yp otheses FDR w ould measure the a v erage probabilit y that it is a false rejection. Use of FDR con trolling pro cedures allo ws for greater leew a y in terms of minor errors in high dimensional settings. In the same pap er men tioned ab o v e, Benjamini and Ho c h b erg (1995) also demonstrated FDR con trol for their step-up pro cedure for indep enden t test statistics no w kno wn as the Benjamini-Ho c h b erg pro cedure; it uses Equation 1.2.2 with (1.2.6) i =i=m; whic h w e refer to as BH st yle cutos. Benjamini and Liu (1999) in tro duced FDR con trol for step-do wn pro cedures using Equation 1.2.3 with some w ell c hosen increasing sequence ( i ) m i=1 . They demonstrated 1.2. INTR ODUCTION AND PRIOR W ORK 11 that under indep endence, the sequence (1.2.7) i = 1 1 1^ m mi + 1 1=(mi+1) i2f1;:::;mg w ould con trol FDR at lev el , where^ is understo o d to the the minim um op erator; w e will refer to these t yp es of cutos as BL-st yle, short for Benjamini-Liu. They compared their pro cedure against the step-up pro cedure, again under indep endence and found that eac h has its adv an tages. Storey (2002) in tro duced a v arian t of FDR called the P ositiv e F alse Disco v ery Rate, or pFDR: (1.2.8) pFDR =E [V=RjR> 0]: Dened to b e the conditional exp ectation of the FDP when at least one n ull h yp othesis is rejected, pFDR further hones in on the notion that our error metric should help the user to gauge their trust in a rejection. Whereas FWER conditions on the unkno wn ground truth, FDR conditions on the kno wn rejection status, ho w ev er it still tak es in to accoun t the irrelev an t scenario in whic h nothing is rejected b y attac hing p ositiv e mass to that case. pFDR impro v es up on that b y only considering the probabilit y of a rejection b eing a false rejection in the case where there is at least one rejection, using the conditional distribution o v er V and R giv en that R > 0, whic h attac hes no mass to the rejection-free realization. Storey (2003) also presen ts an in teresting analysis of pFDR from a Ba y esian p ersp ectiv e. In what follo ws, w e extend our FDR con trolling pro cedures to include metho ds for con trolling pFDR as w ell. F or all of the t yp e 1 error metrics discussed ab o v e, there exist equiv alen t t yp e 2 error metrics. Dene the n um b er of false acceptances to b e V 0 =m 1 (RV ) and the total n um b er of acceptances to b e R 0 =mR. Then the F alse Non-Disco v ery Prop ortion (FNP), F alse Non-Disco v ery Rate (FNR), and P ositiv e F alse Non-Disco v ery Rate (pFNR) are dened to b e FNP =V 0 = (R 0 _ 1) (1.2.9) FNR =E [V 0 = (R 0 _ 1)] (1.2.10) pFNR =E [V 0 =R 0 jR 0 ]: (1.2.11) In a w ell kno wn sequel to the Benjamini-Ho c h b erg pap er, Benjamini and Y ekutieli (2001) sho w ed that the Benjamini-Ho c h b erg (abbreviated as BH) pro cedure also con trolled FDR under the m uc h mild p ositiv e correlation assumption. Finner et al. (2009) presen t a greatly simplied pro of of FDR con trol in a step-up pro cedure giv en a further relaxed correlation condition. Guo and Rao (2008) expand on this result for b oth step-up and step-do wn pro cedures b y b ounding FDR without an y assumptions on the join t distribution using 1.2. INTR ODUCTION AND PRIOR W ORK 12 a clev er optimization framew ork. The ma jorit y of what follo ws in this c hapter amoun ts to an extension of their framew ork to the sequen tial setting. T o understand what is mean t b y con trol of an error metric C, tak eP to b e some class of distributions, C to b e some random error metric, and T to b e some statistical rejection pro cedure suc h that C (T;P ) = E P [C (T )]; here E P for some P2P is the exp ectation under that distribution, andC (T ) to b e the random v alue of the error metric ac hiev ed when applied to the outcome from the test. Then to sa y T con trols C at lev el for all P2P means that8P2P; E P [C (T )] . In the settings fo cused on in this c hapter, w e ha v e no restrictions on P , but later c hapters will examine the case where certain test statistics satisfy a prop ert y referred to as Positive R e gr ession Dep endency on e ach one fr om a subset (referred to as PRDS and discussed in detail in Chapter 2) whic h generalizes p ositiv e correlation. Dene 1 () to b e the indicator function, taking the v alue 1 when the condition is satised and 0 when it is not. Here w e fo cus on C = FDR andC = FDP, but w e discussC = FWER whic h usesC = 1 (V > 0). T w o further examples are C = -FDP, whic h usesC = 1 (FDP ) for 2 (0; 1), andC =k -FWER whereinC = 1 (V k) for km, ho w ev er w e will not explore their con trol in this thesis. 1.2.0.2. Se quential T esting: W ald and the SPR T. The eld of sequen tial testing w as pioneered in the mid-1940’s b y Abraham W ald (1973). He in tro duced the original Sequen tial Probabilit y Ratio T est, or SPR T, whic h is b oth the motiv ation for and b est example of our additions. SPR T s are a class of tests of simple vs.simple h yp otheses in the sequen tial setting that terminate once the log lik eliho o d ratio, abbreviated hereafter as LLR, crosses an upp er or lo w er barrier, rejecting or accepting the n ull resp ectiv ely . W ald’s appro ximation allo ws for the explicit computation of cuto v alues based on t yp e 1 error and t yp e 2 error b ounds for SPR T s, as elab orated up on in Go vindara julu (1975). SPR T s require t w o cuto v alues A > 0 > B , the join t densit y of the rst t samples under the n ull f t 0 (x 1 ;x 2 ;:::;x t ), and their join t densit y under the alternativ e f t 1 (x 1 ;x 2 ;:::;x t ). Then the SPR T test statistic is (1.2.12) (t) := log f t 1 (x 1 ;x 2 ;:::;x t )=f t 0 (x 1 ;x 2 ;:::;x t ) : A t eac h time step t, if (t)A then the n ull is rejected and the exp erimen t terminated; if (t)B then the n ull is accepted and the exp erimen t terminated; nally , if A> (t)>B then more data is required. As an example consider the scenario where X t iid P oisson () withH 0 : = 0 andH 1 : = 1 . Then the probabilit y mass function (pmf ) of X t is f (x) = x e =x! 8x2f0; 1; 2;:::g: 1.2. INTR ODUCTION AND PRIOR W ORK 13 Th us, the LLR statistic tak es the form (t) = log t Y s=1 Xs 1 e 1 =X s ! ! log t Y s=1 Xs 0 e 0 =X s ! ! = t X s=1 1 +X s log 1 + 0 X s log 0 c = ( 1 0 )t + log ( 1 = 0 ) t X s=1 X s : If instead w e use X t iid Binomial (n;p) with H 0 :p =p 0 and H 1 :p =p 1 , then the pmf of X t is f p (x) = n x p x (1p) nx and the test statistic tak es the form (t) = log t Y s=1 n X s p Xs 1 (1p 1 ) nXs ! log t Y s=1 n X s p Xs 0 (1p 0 ) nXs ! = t X s=1 n log (1p 1 ) +X s log p 1 1p 1 +n log (1p 0 )X s log p 0 1p 0 = log 1p 1 1p 0 nt + log p 1 (1p 0 ) p 0 (1p 1 ) t X s=1 X s : Our goal is to pro vide the t yp e 1 and t yp e 2 error con trols (1.2.13) P H0 (rejectH 0 ) =P H0 (9t s.t. (t)A\8t 0 <t (t 0 )>B) (1.2.14) P H1 (acceptH 0 ) =P H0 (9t s.t. (t)B\8t 0 <t (t 0 )<A): W ald (1973) pro vides an astoundingly con v enien t metho d for doing so for an y simple vs. simple (meaning that eac h h yp othesis completely sp ecies one, and only one, distribution) SPR T: A = log 1 + B = log 1 1.3. MAIN RESUL TS 14 where is a small, p ositiv e adjustmen t constan t. These are discussed in greater detail in the App endix A.1.1. The form of our con trol inequalities imply that a high v alue of the test statistic p oin ts to w ards the rejection of the n ull, while a lo w v alue suggests acceptance, ho w ev er this is not a required. 1.2.0.3. Se quential T esting of Multiple Hyp otheses. Bartro and Lai (2010) and Bartro and Song (2014; 2015) made great strides in com bining these t w o disciplines, in tro ducing a FDR con trolling Sequen tial BH pro cedure, and a n um b er of FWER con trolling sequen tial pro cedures. In an earlier pap er Bartro and Song (2013) also discuss sequen tial GLR tests that allo w for comp osite h yp otheses, and a normal appro ximation of their distributions that allo ws for explicit cuto computation, with explicit details in Bartro et al. (2012, p85:87). Bartro (2014) extended this w ork to other error measures including -FDP, the probabilit y that the ratio of incorrectly rejected n ulls to total rejections exceeds , andk -FWER, the probabilit y that the total n um b er of false rejections exceeds k . De and Baron(2012; 2015) and Song and F ellouris (2016; 2017) ha v e also explored FWER and k -FWER con trolling sequen tial pro cedures, but with sync hronous stopping times, meaning that all streams m ust terminate sim ultaneously . This requires totally dieren t stopping criteria and analysis metho ds. Ja v anmard and Mon tanari (2017) presen t a FDR-exceedance con trolling pro cedure for sequen tial testing of m ultiple h yp otheses wherein the h yp otheses themselv es are deliv ered sequen tially , but m ust b e decided up on immediately using only the data that has arriv ed b y that time. This diers from the settings of the other pap ers discussed here in that they assume a xed n um b er of kno wn h yp otheses, rev ealed to the user at the start of the pro cedure, where data is deliv ered sequen tially and the p oin t at whic h decisions are made ab out the h yp otheses are random functions of the data and the pro cedure. 1.3. Main Results 1.3.1. Setup and Notation. In what follo ws w e consider the set of n ull h yp otheses H 1 0 ;H 2 0 ;:::;H m 0 , to b e subsets of their resp ectiv e parameter spaces 1 ; 2 ;:::; m , and understand that the n ull is true when the true parameter i 2H i 0 . W e will use the notationP to denote the join t distribution o v er all streams where the parameters v alues are = ( i ) i2f1;:::;mg 2 := 1 2 ::: m ; in a sligh t abuse of notation, w e will use P 0 to denote the true distribution from whic h the data is generated. In scenarios in whic h w e wish to con trol t yp e 2 error in addition to t yp e 1 error, w e will also consider the set of alternativ e h yp otheses H 1 1 ;H 2 1 ;:::;H m 1 , again considered to b e subsets of the full parameter spaces and disjoin t from the subsets dened b y the resp ectiv e n ull h yp otheses. In order to use some of the adv anced mac hinery pro vided b y W ald’s w ork on sequen tial testing w e ma y sometimes require that the h yp otheses b e simple vs. simple. F or eac h of the m n ull h yp othesis, w e receiv e data from their resp ectiv e streams (X 1 1 ;X 1 2 ;:::;X 1 t ;:::);:::; (X m 1 ;X m 2 ;:::;X m t ;:::). Some of these data streams ma y b e o v erlapping or iden tical; some ma y b e m ultidimensional or ev en v ary in dimension o v er time. A t eac h time step t; whic h need not b e an actual temp oral time step, w e compute 1.3. MAIN RESUL TS 15 the test statistics 1 (t);:::; m (t) from eac h stream that has not y et terminated, using only the data from their resp ectiv e streams a v ailable at time t; for eac h i2f1;:::;mg; t 1 eac h statistic i (t) is computed as a function of the data p oin ts X i 1 ;X i 2 ;:::;X i t and only those date p oin ts. W e then test these statistics to determine whether w e can reject H i 0 in fa v or of H i 1 ; accept H i 0 , or if w e need to con tin ue collecting data for the i th h yp othesis pair. Without loss of generalit y , w e assume the rst m 0 n ull h yp otheses are true and that the nal m 1 n ull h yp otheses are false where m =m 0 +m 1 , though the user do es not kno w m 0 ; m 1 ; or the ordering; this will greatly simplify our analysis without compromising its generalit y . Algorithm 1 Finite Horizon, Rejectiv e Sequen tial Step-do wn Pro cedure 1: pro cedure ((X 1 ;:::;X m ); (A m ;:::;A 0 );T ) 2: k 1 . Highest signicance lev el not y et tripp ed 3: t 0 4: I f1;:::mg . Set of curren tly activ e h yp otheses 5: while t<T\k<m do . Collect data un til horizon is reac hed or all h yp otheses are rejected 6: t t + 1 7: i (t) i (X i 1:t ) 8i2I 8: if9i2I s.t. i (t)>A k then . If an y activ e h yp othesis trips the most signican t remaining cuto, initiate rejections. Otherwise con- tin ue sampling. 9: k t 0 . Rejections at curren t time step 10: i ? arg max i2I i (t) 11: while k t mk \ i ? (t)>A k+kt do . Step do wn through signicance lev els 12: I Infi ? g . Reject most signican t h yp othesis 13: k t k t + 1 . Incremen t n um b er of h yp otheses rejected during this time step 14: i ? arg max i2I i (t) . Up date most signican t activ e h yp othesis 15: end while 16: k k +k t 17: end if 18: end while 19: A ccept I 20: Rejectf1;:::;mgnI 21: end pro cedure 1.3. MAIN RESUL TS 16 Algorithm 2 Innite Horizon, A cceptiv e-Rejectiv e Sequen tial Step-do wn Pro cedure 1: pro cedure ((X 1 ;:::;X m ); (A m ;:::;A 0 ); (B m ;:::;B 0 )) 2: k r 1, k a 1 . Highest rejection/acceptance signicance lev el not y et tripp ed. 3: t 0 4: I f1;:::mg . Set of curren tly activ e h yp otheses. 5: I r ,I a . Set of curren tly rejected/accepted h yp otheses. (Empt y) 6: while k r +k a <m do . Collect data un til all h yp otheses ha v e b een accepted or rejected. 7: t t + 1 8: i (t) i (X i 1:t ) 8i2I 9: if9i2I s.t. i (t)>A k r then . If an y activ e h yp othesis trips the highest remaining rejection cuto, initiate rejections. 10: k r t 0 . Rejections at curren t time step. 11: i ? arg max i2I i (t) 12: while k r t mk r k a \ i ? (t)>A k+k r t do . Step do wn through signicance lev els. 13: I Infi ? g . Reject most signican t h yp othesis. 14: I r I r [fi ? g 15: k r t k r t + 1 . Incremen t n um b er of h yp otheses rejected during this time step. 16: i ? arg max i2I i (t) . Up date most signican t activ e h yp othesis. 17: end while 18: k r k r +k r t . Up date most signican t rejection lev el not y et tripp ed. 19: end if 20: if9i2I s.t. i (t)<B k a then . Rep eat same pro cedure for acceptances. 21: k a t 0 . A cceptances at curren t time step. 22: i ? arg min i2I i (t) 23: while k a t mk r k a \ i ? (t)<B k+k a t do . Step do wn through signicance lev els 24: I Infi ? g . A ccept most signican t h yp othesis. 25: I a I a [fi ? g 26: k a t k a t + 1 . Incremen t n um b er of h yp otheses accepted during this time step. 27: i ? arg min i2I i (t) . Up date most signican t activ e h yp othesis. 28: end while 29: k a k a +k a t . Up date most signican t acceptance lev el not y et tripp ed. 30: end if 31: end while 32: Reject I r , A ccept I a 33: end pro cedure Here w e presen t t w o classes of pro cedures. The rst uses a nite sampling horizon, with only rejections o ccurring un til that horizon is reac hed. The second allo ws for innite sampling, and b oth rejections and acceptances at an y time step. Dene the rejection and acceptance (when relev an t) cutos to b e (1.3.1) 1 =A 0 A 1 :::A m 0 (1.3.2) 1 =B 0 B 1 :::B m 0: A t eac h step w e p erform a step-do wn test for either rejection or for b oth acceptance and rejection, remo ving an y h yp otheses and tripp ed signicance cutos from the follo wing steps. These terminations tak e the form of a xed-sample step-do wn where the cutos are the remaining, un-tripp ed cutos and the statistics are 1.3. MAIN RESUL TS 17 the curren t v alues of sequen tial statistics of all activ e h yp otheses. Once a cuto has b een tripp ed, it is remo v ed from subsequen t consideration, as A 1 is at t = 5 in Figure 1.3.1. This mak es mak es the criteria for later rejections (or acceptances) less stringen t than those of the earlier terminations. The rationale b ehind this is that once w e ha v e rejected a n um b er of n ull h yp otheses with high signicance w e exp ect that they w ere correctly rejected, this giv es us more exibilit y to reject h yp otheses with w eak er evidence while still main taining FDR con trol. A nite horizon rejectiv e v ersion, where an ything not rejected b y the nite horizon T is assumed to b e accepted, is sp elled out in full in Algorithm 1. Its extension to the acceptiv e-rejectiv e v arian t, wherein eac h h yp othesis m ust b e explicitly accepted or rejected via a sequen tial step-do wn rule b efore the algorithm terminates, is sp elled out in full in Algorithm 2 and illustrated in Figure 1.3.1. The main dierence is the replacemen t of the nite horizon c hec king condition on the main lo op (and horizon time acceptance of all activ e h yp otheses) with an acceptance c hec k at eac h time step that mirrors the rejection c hec k. The order do es not matter b ecause no h yp othesis that migh t trip a rejection lev el can ha v e a statistic less than 0, whic h means it could not trip an acceptance lev el, and vice v ersa. W e replace the p-v alues with general test statistics in order to allo w generalization of the routine from rejectiv e only to acceptance and rejection. A dditionally this p ermits us to use all the mac hinery dev elop ed around SPR T s. 1.3. MAIN RESUL TS 18 Figure 1.3.1. An example of a sequen tial step-do wn pro cedure. The paths are LLRs for simple vs.simple h yp otheses ab out the P oisson rate. The annotations are of the form: Hyp othesis n um b er (termination t yp e at termination step) ground truth. F ain t lines sho w the future paths the statistics w ould ha v e tak en if they had not terminated. 1.3. MAIN RESUL TS 19 Figure 1.3.2. Sample LLR paths for Gaussian data. Finite horizon rejectiv e. Hyp otheses 0 and 1 did not terminate b efore the nite horizon and are therefore accepted . In b oth the rejectiv e and acceptiv e-rejectiv e v arian ts of our sequen tial pro cedure, tak e i for i 2 f1;:::;mg to b e the random time at whic h a n ull h yp othesis is either accepted or rejected, and there- fore no longer sampled. These are dened implicitly in Algorithms 1 and 2; explicit denitions are pro vided in App endix A.2. In the nite horizon rejectiv e case w e set a time limit T , testing for rejection only at eac h time step b efore T; and accepting an y n ull h yp otheses that surviv e to the end without ha ving b een rejected; no acceptances o ccur b efore time T . In this case for all i2f1;:::;mg P ( i T ) = 1, and i <T implies rejection, while i =T implies acceptance. As discussed ab o v e, the innite horizon acceptiv e- rejectiv e case refers to a testing pro cedure that con tin ues on un til all pairs of h yp otheses ha v e b een decided. In theorems regarding the innite horizon, acceptiv e-rejectiv e pro cedure w e will assume termination, i.e. P (8i2f1;:::;mg i <1) = 1. This ma y b e violated if particularly pathological test statistics are em- plo y ed, for instance i (t) = 0, in whic h case the pro cedure nev er terminates and no nal FDR or FNR can b e dened. 1.3.2. Main Theorems and Corollaries. Our main results expand up on and generalize the w ork of Guo and Rao (2008). In their pap er they b ound the maximal FDR ac hiev able b y a xed-sample step-do wn pro cedure under an y constrain ts limited to the marginal distributions, as long as the p-v alues of the true n ull 1.3. MAIN RESUL TS 20 h yp otheses are uniformly distributed. This is a stronger condition than p-v alue v alidit y , in that it pac ks the maximal amoun t of probabilit y mass in to the lo w er v alues (indicativ e of rejection) allo w able sub ject to the denition of a p-v alue. Only a minor adaptation is required to generalize their results to an y statistic with a v alid p-v alue. Here w e further extend the generalized v ersion to the innite horizon acceptiv e-rejectiv e sequen tial case, the nite horizon rejectiv e sequen tial case, and to the pFDR and pFNR v arian ts of the rst t w o cases. Finite Horizon, Rejectiv e Innite Horizon, A cceptiv e-Rejectiv e FDR (and FNR) Theorem 1.1 Theorem 1.2 pFDR (and pFNR) Theorem 1.3 Theorem 1.4 T able 1. Directory of Error Con trol Theorems The error b ounds in the w ork w e presen t here tak e the form of a function D (~ ), dened b elo w in Equation 1.3.4, that is linear in ~ , where ~ is the v ector of marginal probabilit y b ounds in the con trol inequalities similar to those in Equation 1.2.13, but generalized to the m ultiple h yp othesis setting. Emplo ying the linearit y of D , if w e wish to con trol FDR at some scalar lev el 2 (0; 1) using a p-v alue cuto v ector (hereafter referred to as the cuto v ector, to a v oid confusion with other p sym b ols in tro duced later) with the shap e of some v ector ~ , w e create a new v ector (1.3.3) ~ 0 = ~ D (~ ) ; and use that to calculate the statistic cutos A j and B j . By shap e w e mean 8i;j2f1;:::;mg the ratios satisfy i = j = 0 i = 0 j . In general, these b ounds can b e impro v ed when the n um b er m 0 of true n ull h yp otheses is kno wn a priori. In this case, with m 1 =mm 0 the n um b er of false n ull h yp otheses, the factor D (~ ) can b e replaced b y the smaller quan tit y (1.3.4) D(m 0 ;m 1 ;~ ) =m 0 0 @ m1+1 X j=1 j j1 j + m X j=m1+2 m 1 ( j j1 ) j(j 1) 1 A ; th us increasing the p o w er of the test. In the more common case where this balance is unkno wn to the user, w e simply tak e the maxim um o v er these functions. With sligh t abuse of notation, dene (1.3.5) D (~ ) = max 1m0m m1=mm0 D(m 0 ;m 1 ;~ ): 1.3. MAIN RESUL TS 21 In Figure 1.3.3 w e presen t the Guo-Rao scaling factor for BH-st yle cutos with resp ect to the total n um b er of h yp otheses, b ecause BH st yle cutos allo w for a clean example of Equations 1.3.4 and 1.3.5, written out explicitly at the end of this section; initially dev elop ed for use with step-up pro cedures in that they con trol FDR at the nominal lev el when the streams are indep enden t (Benjamini and Ho c h b erg, 1995), Finner et al. (2009) demonstrated that they con trol FDR of b oth step-up and step-do wn pro cedures under a certain class of dep endency structures, whic h includes the case of indep enden t h yp otheses. These plots are th us visualizations of D (i=m) i2f1;:::;mg = =D ((1=m; 2=m;:::; 1)). Giv en that the scaling factor is appro ximately logarithmic in terms of m, w e nd that ~ = (i= (m logm)) i2f1;:::;mg should con trol FDR at lev el . In Figure 1.3.4 w e presen t t w o plots analyzing the scaling factor when BL-st yle cutos are used. Recall that the shap e of BL-st yle cutos dep ends b oth on the n um b er of h yp otheses and the lev el at whic h the user wishes to con trol FDR when the streams are indep enden t, and w ere dev elop ed to con trol FDR of step-do wn pro cedures under indep endence. Eac h color in these plots corresp onds to the lev el of the indep enden t, unscaled BL cutos. In the left hand plot, w e compute the Guo-Rao FDR b ound on the BL cutos then divide b y the con trol lev el for indep endence. In the righ t hand plot, w e do the same thing but divide again b y the total n um b er of h yp otheses. In b oth plots, w e see that these ratios do not dep end on the FDR con trol lev els, in spite of their eect on the shap e of the cuto curv es, and that as m gro ws large the scaling for BL st yle cutos is appro ximately m=4. The reader ma y note that the n um b er of h yp otheses at whic h the scaling factors are computed diers b et w een the initial lev els. This is b ecause for m 1= BL st yle cutos saturate, with at least one of the least signican t i ’s b eing equal to 1, therefore w e only test m< 1=. 1.3. MAIN RESUL TS 22 Figure 1.3.3. FDR scales linearly with the cuto v ector. Using BH st yle cutos with an unkno wn n um b er of true n ull h yp otheses, the scale is close to logarithmic in the total n um b er of h yp otheses. The y-axis is the scaling factor max m0m D(m 0 ;m m 0 ; (i=m) m i=1 )=. 1.3. MAIN RESUL TS 23 Figure 1.3.4. The Guo-Rao ination for BL-st yle cutos app ears not to dep end up on the initial lev el. It seems to con v erge to m=4 as m b ecomes large. Num b er of Hyp otheses 10 29 85 146 251 429 1258 3686 10797 18478 31622 Scaling F actor 1.840 2.575 3.414 3.856 4.310 4.766 5.700 6.652 7.617 8.103 8.592 T able 2. FDR scaling factor under BH cutos Here w e presen t the t w o theorems cen tral to our w ork. First, con trol of FDR in the nite horizon, rejectiv e sequen tial step-do wn pro cedure. Second, con trol of b oth FDR and FNR and the innite horizon, acceptiv e-rejectiv e step-do wn pro cedure. Theorem 1.1. Finite Horizon R eje ctive FDR Contr ol The nite horizon, r eje ctive se quential step-down pr o c e dur e describ e d in A lgorithm 1 with cutos as in Equation 1.2.1 and cutos A j as sp e cie d in Equation 1.3.1 satisfying (1.3.6) 8im;2H i 0 P (9t<T s.t. i (t)A j ) j 1.3. MAIN RESUL TS 24 pr ovides the typ e 1 err or b ound FDRD (~ ) under the true distribution, r e gar d less of the dep endenc e structur e, wher e D (~ ) is dene d as in Equation 1.3.5. The innite horizon v arian t is similar, but requires sligh tly dieren t marginal inequalities and pro vides t yp e 2 error con trol as w ell. Theorem 1.2. Innite Horizon FDR and FNR Contr ol The innite horizon, ac c eptive-r eje ctive se quential step-down pr o c e dur e describ e d in A lgorithm 2 with ~ and ~ satisfying Equation 1.2.1 (and an e quivalent for typ e 2 err or) and A j ’s and B j ’s as in Equations 1.3.1 and 1.3.2 satisfying (1.3.7) 8im; jm;2H i 0 P (9t<1 s.t. i (t)A j \8t 0 <t (t 0 )>B 1 ) j (1.3.8) 8im; jm;2H i 1 P (9t<1 s.t. i (t)B j \8t 0 <t i (t 0 )<A 1 ) j pr ovides the fol lowing b ounds on typ e 1 and typ e 2 err or under the true distribution, r e gar d less of the dep en- denc e structur e: FDRD(~ ) FNRD( ~ ): A gain, D (~ ) is dene d as in Equation 1.3.5, and D ~ is the same function applie d to the typ e 2 mar ginal c onstr aints ve ctor. These theorems allo w for construction of FDR and FNR con trolling pro cedures via the scaling approac h describ ed in Equation 1.3.3. In the case of an SPR T, the cutos can then b e used to construct the statistic cutos using the m ultiple h yp othesis W ald appro ximations discussed in Section A.1.2. W e also presen t similar theorems for pFDR and pFNR con trol, and corollaries for application of it to SPR T and more general tests. 1.3. MAIN RESUL TS 25 Theorem 1.3. Finite Horizon R eje ctive pFDR Contr ol The nite horizon, r eje ctive se quential step-down pr o c e dur e describ e d in A lgorithm 1 with cutos as in Equation 1.2.1 and test statistic cutos A j as sp e cie d in Equation 1.3.1 satisfying the mar ginal c onstr aints in Equation 1.3.6 pr ovides the typ e 1 err or b ound under the true distribution with m 0 true nul l hyp otheses and m 1 false nul l hyp otheses, r e gar d less of the dep endenc e structur e: pFDRD (m 0 ;m 1 ;~ =P (R> 0)) =D (m 0 ;m 1 ;~ )=P (R> 0): F urther, we also have pFDR D (m 0 ;m 1 ;~ ) max 1im P (9t<T s.t. i (t)A 1 ) D (m 0 ;m 1 ;~ ) min 1im P (9t<T s.t. i (t)A 1 ) : In the nite horizon, rejectiv e pro cedure, the quan tit y P (R> 0) is equiv alen t to P 9i2f1;:::;mg; t<T s.t. i (t)A 1 ; and dep ends on the pro cedure and cutos. The implications of this theorem on determining the actual lev el of pFDR con trol are discussed in Section A.1.4. Theorem 1.4. pFDR and pFNR Contr ol for Innite Horizon The innite horizon, ac c eptive-r eje ctive se quential step-down pr o c e dur e describ e d in A lgorithm 2, with ~ and ~ satisfying Equation 1.2.1 (and an e quivalent for typ e 2 err or) and A j ’s and B j ’s as in Equations 1.3.1 and 1.3.2 satisfying the mar ginal c onstr aints in Equations 1.3.7 and 1.3.8, pr ovides the typ e 1 and typ e 2 err or b ounds under the true distribution with m 0 true nul l hyp otheses and m 1 false nul l hyp otheses, r e gar d less of the dep endenc e structur e: pFDRD m 0 ;m 1 ; ~ P (R> 0) D (m 0 ;m 1 ;~ ) max 1im P (9t<1 s.t. i (t)A 1 ;8t 0 <t i (t 0 )>B m ) D (m 0 ;m 1 ;~ ) P (9t<1 s.t. i (t)A 1 ;8t 0 <t i (t 0 )>B m ) 8i2f1;:::;mg; 1.3. MAIN RESUL TS 26 pFNRD m 1 ;m 0 ; ~ P (R 0 > 0) ! D m 1 ;m 0 ; ~ max 1im P (9t<1 s.t. i (t)B 1 ;8t 0 <t i (t 0 )<A m ) D m 1 ;m 0 ; ~ P (9t<1 s.t. i (t)B 1 ;8t 0 <t i (t 0 )<A m ) 8i2f1;:::;mg: Note that in the case of a simple vs. simple SPR T with b oth m 1 > 0 andm 0 > 0, w e ma y apply W ald’s appro ximation to the denominator of the b ounds ab o v e P H i 0 (9t<1 s.t. i (t)B 1 ;8t 0 <t i (t 0 )<A m ) = 1P H i 0 ( 8t<1 i (t)>B 1 [ 9t<1 s.t. i (t)A m ;8t 0 <t i (t 0 )>B 1 ) = 1P H i 0 (9t<1 s.t. i (t)A m ;8t 0 <t i (t 0 )>B 1 ) 1 m : Using this appro ximate equiv alence and w e ma y then ac hiev e the follo wing appro ximate b ounds, sub ject to the conditions in Theorem 1.4: pFDR. D(m 0 ;m 1 ;~ ) 1 m pFNR. D(m 1 ;m 0 ; ~ ) 1 m ; where. is understo o d to mean less than or on the order of. F or arbitrary test statistics, w e ma y add p o w er conditions requiring the existence of a lo w er b ound, the probabilit y that a false (true) n ull is correctly rejected (accepted) at the most extreme lev el, and th us obtain the follo wing more general b ounds: Cor ollar y 1.1. pFDR and pFNR Contr ol for Innite Horizon with Gener al T est Statistics In addition to the assumptions pr esente d in The or em 1.4, given the fol lowing lower-b ounds on ac cur ate r eje ction and ac c eptanc e8im: 9i2f1;:::;mg; s.t. 82H i 1 P (9t<1 s.t. i (t)A 1 ; i (t 0 )>B m 8t 0 <t) 1 1.3. MAIN RESUL TS 27 9i2f1;:::;mg; s.t. 82H i 0 P (9t<1 s.t. i (t)B 1 ; i (t 0 )<A m 8t 0 <t) 0 pr ovides the typ e 1 and typ e 2 err or b ounds under the true distribution with m 0 true nul l hyp otheses and m 1 false nul l hyp otheses, r e gar d less of the dep endenc e structur e pFDR D(m 0 ;m 1 ;~ ) 1 pFNR D(m 1 ;m 0 ; ~ ) 0 : 1.3.3. Pro ofs. T o pro v e the ab o v e theorems w e will b egin b y in tro ducing to ols to analyze a purely rejectiv e, nite horizon scenario, then use them to pro v e Theorem 1.1 in Section 1.3.3.4. These to ols con- sist of lemmas dening sets of sym b ols that allo w us to translate join t distributions of test statistics in to decomp ositions of FDR, and related error metrics. Once w e ha v e pro v en the rst theorem, w e will in tro duce v ariations of the sym b ols, lemmas, and decomp ositions that apply to the nite horizon pFDR con trolling case, as in Theorem 1.3, in Section 1.3.3.5. Finally , in Section 1.3.3.6, w e address the innite horizon v ari- an ts, Theorems 1.2 and 1.4, and explain an y minor mo dications of the pro of of Theorem 1.1 necessary to incorp orate the mo dications and pro v e the relev an t theorem. In the pro of of the main theorems, w e tak e the suprem um of FDR (or the relev an t error metric) o v er v alid join t distributions b y emplo ying the decomp ositions pro vided b y the lemmas as w ell as the restrictions of the v alues of certain sym b ols in Lemma 1.4, and sho w that this optimization problem can b e relaxed to one with an analytic solution. 1.3.3.1. T o ols for A nalyzing FDR. Again taking the rejection cutos A j to b e as in Equation 1.3.1, dene the probabilit y of rejecting the i th n ull h yp othesis at lev el j when exactly k total rejections o ccur as (1.3.9) p ijk =P ( i ( i )2 [A j ;A j1 ); i <T; R =k); where i (t) is the v alue of the test statistic for the i th h yp othesis at time t, T is the maxim um n um b er of samples (i.e. the nite horizon), R is the total n um b er of rejections, and i is the time step at whic h the i th h yp othesis is rejected. Using these sym b ols, w e in tro duce our rst decomp osition of FDR in the follo wing lemma. Lemma 1.1. FDR p Symb ol De c omp osition 1.3. MAIN RESUL TS 28 F or the nite horizon, r eje ctive step-down pr o c e dur e describ e d in A lgorithm 1, using cutos A j as in Equation 1.3.1 and p ijk as in 1.3.9, wher e the rst m 0 nul l hyp otheses ar e true and the r emaining m 1 = mm 0 ar e false, we may de c omp ose FDR as fol lows (1.3.10) FDR = m0 X i=1 m X j=1 m X k=j 1 k p ijk : Pr oof. FDR =E[V=(R_ 1)] = m X k=1 E[VjR =k] k P (R =k) = m X k=1 m0 X i=1 P (H i 0 rejected; R =k) k = m0 X i=1 m X k=1 k X j=1 P (H i 0 rejected at lev elj; R =k) k = m0 X i=1 m X j=1 m X k=j 1 k p ijk Using this lemma w e can see that the true FDR is b ounded b y the taking the follo wing suprem um o v er all join t distributions of statistics satisfying the marginal conditions for h yp otheses i m 0 , the true n ulls. Here w e tak e the suprem um o v er all suc h join t distributions, sub ject to the marginal constrain ts (1.3.11) FDR 1 := sup m0 X i=1 m X j=1 m X k=j 1 k p ijk s:t: 8im 0 ; Jm; P ( i <T; i ( i )A J ) = J X j=1 m X k=j p ijk J : Clearly , all p ijk 0 andfp ijk g i;j;k m ust b e the result of passing some join t distribution of test statistics through our pro cedure. It is v ery imp ortan t to note that b oth f i g i2f1;:::;mg and R dep end on the rejection pro cedure in question, and are the conduit through whic h this optimization dep ends on the pro cedure in that they limit the space of v alues attainable b y fp ijk g i;j;k . In the last expression, FDR = FDR (P 0 ) is actually a function of the true join t distribution P 0 of test statistics, and eac h p ijk = p ijk (P ) is actually a function of the join t distribution P as w ell as the cuto v alues A : A v ersion of Equation 1.3.11 sho wing these distributional dep endencies is 1.3. MAIN RESUL TS 29 FDR(P 0 ) 1 = sup P2P m0 X i=1 m X j=1 m X k=j 1 k p ijk (P ); whereP is the space of all join t distributions of test statistics, sub ject to the marginal conditions on the true n ulls, whose true n ulls matc h those in P 0 . Note that the probabilit y of a true n ull h yp othesis b eing rejected b efore the end of the test at lev el j is b ounded as follo ws: 8im 0 ; jm; P ( i <T; i ( i )A j )P H i 0 (9t<T s.t. i (t)A j ) b ecause i <T; i ( i )A j 9t<T s.t. i (t)A j and P 9t<T s.t. i (t)A j =P H i 0 9t<T s.t. i (t)A j : Th us giv en the marginal b ounds for all of the n ull h yp otheses 8im; jm; P H i 0 (9t<T s.t. i (t)A j ) j ; w e ma y b ound individual false rejections for all P2P: 8im 0 ; jm; P ( i <T; i ( i )A j ) j : 1.3.3.2. R eje ction Pr o c e dur e as a F unction of the Joint Distribution: Permutation Exp ansion. T o enforce the restriction of our p ijk v alues to those that could result from the application of our rejection pro cedure to a v alid join t distribution, w e m ust further expand our dictionary of sym b ols. W e b egin b y dening sym b ols to denote the probabilit y of sp ecic p erm utations of false rejections, their corresp onding lev els, and the total n um b er of rejections as follo ws: (1.3.12) q (k) (i1;j1):::(i ` ;j ` ) =P (R =k;8d` i d <T\ i d ( i d )2 [A j d ;A j d 1 ); 8i2f1;:::;m 0 gnfi 1 ;:::;i ` g i =T ) i.e., the probabilit y that exactly k rejections o ccurred, of whic h exactly ` w ere false rejections and those false rejections w ere indexed b y the sp ecied i d ’s whic h terminated in corresp onding rejection in terv als [A j d ;A j d 1 ). Again, w e ma y think of these sym b ols as functions of the join t distribution q (k) (i1;j1):::(i ` ;j ` ) (P ). W e wish to translate the optimization problem in to one in v olving the q () (;) ’s, so w e in tro duce notation represen ting relev an t sets of q () (;) ’s o v er whic h our sums will b e tak en: 1.3. MAIN RESUL TS 30 (1.3.13) (k) ` :=f((i 1 ;j 1 );:::; (i ` ;j ` )) : 1i 1 <:::<i ` m 0 ; 1j 1 ;j 2 ;:::;j ` kg (1.3.14) (k) ` (i;j) := n ((i 1 ;j 1 );:::; (i ` ;j ` ))2 (k) ` : (i;j)2f(i 1 ;j 1 );:::; (i ` ;j ` )g o : So (k) ` is the set of all `-v ectors whose comp onen ts are pairs of indices of true n ulls and their corresp onding rejection lev els when exactly k total h yp otheses are rejected. W e ma y cap the j ’s atk b ecause our pro cedure do es not p ermit rejections at lev els less signican t than the total n um b er of rejections; see Algorithm 1. The set (k) ` (i;j) is similarly dened except that it restricts the rst suc h set to the those elemen ts in whic h the i th true n ull is rejected at lev el j . Lemma 1.2. p to q Symb ols for FDR Contr ol (1.3.15) p ijk = k^m0 X l=1 X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) : Pr oof. Theq sym b ols in (k) ` (i;j) represen t disjoin t ev en ts b ecause if an y i d do es not matc h up b et w een t w oq sym b ols then the set of false rejections diers, and if an y j d do esn’t matc h then the lev el at whic h that false rejection o ccurred diers. Therefore, their sum is equiv alen t to the probabilit y of the union of their ev en ts. Because (k) ` (i;j) is, b y denition, all of the sets ((i 1 ;j 1 );:::; (i ` ;j ` ))2 (k) ` that include the pair (i;j) w e ha v e X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) =P i ( i )2 [A j ;A j1 ); i <T; R =k; V =` : Summing o v er ` w e ha v e k^m0 X l=1 X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) = k^m0 X l=1 P i ( i )2 [A j ;A j1 ); i <T; R =k; V =` =P i ( i )2 [A j ;A j1 ); i <T; R =k; V =` =p ijk Here ` is the total n um b er of false rejections, whic h m ust b e greater than or equal to 1 for a true h yp othesis to ha v e b een falsely rejected (whic h is exactly what p ijk represen ts the probabilit y of ), m ust b e 1.3. MAIN RESUL TS 31 less than k b ecause there cannot b e more false rejections than total rejections, and m ust b e less than m 0 b ecause the total n um b er of falsely rejected true n ull h yp otheses cannot exceed the total n um b er of true n ull h yp otheses. Th us these sums include all rejection p erm utations that could con tribute to p ijk , and the sum represen ts a union of non-o v erlapping ev en ts. Plugging Equation 1.3.15 in to the p ortion of the ob jectiv e function of whic h w e are taking the sup in Equation 1.3.11, w e simplify b y follo wing the pro of of Guo and Rao’s (2008) Lemma 3.2: m0 X i=1 m X j=1 m X k=j 1 k p ijk = m0 X i=1 m X j=1 m X k=j k^m0 X `=1 X (k) ` (i;j) 1 k q (k) (i1;j1):::(i ` ;j ` ) = m X k=1 k^m0 X `=1 m0 X i=1 k X j=1 X (k) ` (i;j) 1 k q (k) (i1;j1):::(i ` ;j ` ) = m0 X `=1 m X k=` m0 X i=1 k X j=1 X (k) ` (i;j) 1 k q (k) (i1;j1):::(i ` ;j ` ) = m0 X `=1 m X k=` X (k) ` ` k q (k) (i1;j1):::(i ` ;j ` ) The rst equalit y is simply a result of reordering and reparameterization the sums o v er f(j;k) : 1jkmg. The second equalit y reparameterizesf(k;`) : 1km; 1`k^m 0 g tof(k;`) : 1`m 0 ; `kmg. The nal equalit y is the most complicated, and relies on the fact that for xed k and`; eac hq (k) (i1;j1):::(i ` ;j ` ) o c- curs in exactly` dieren t (k) ` (i;j)s. That is to sa y , for xed k ,`, andq (k) (i1;j1):::(i ` ;j ` ) ,8d2f1;:::;`g q (k) (i1;j1):::(i ` ;j ` ) 2 (k) ` (i d ;j d ) and8 (k 0 ) ` 0 (i 0 ;j 0 ) where either k6=k 0 , `6=` 0 , or (i 0 ;j 0 ) = 2f(i 1 ;j 1 ):::(i ` ;j ` )g then q (k) (i1;j1):::(i ` ;j ` ) = 2 (k 0 ) ` 0 (i 0 ;j 0 ). W e no w translate the optimization problem in to equations in v olving the q (k) ::: ’s, with the suprem um tak en o v er v alid join t distributions: (1.3.16) FDR 1 = sup m0 X `=1 m X k=` X (k) ` ` k q (k) (i1;j1):::(i ` ;j ` ) s:t: 8im 0 ;Jm; J X j=1 m X k=j k^m0 X l=1 X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) J : 1.3.3.3. R elaxation to T r actable Pr oblem. F rom this p oin t on, our pro of of Theorem 1.1 coincides exactly with that of Guo and Rao (2008), as w e ha v e fully addressed the sequen tial nature of our problem. It is p os- sible that Lemma 1.3 migh t allo w for further renemen t in the sequen tial setting, allo wing for impro v emen ts 1.3. MAIN RESUL TS 32 to Lemma 1.4, and therefore all of the theorems presen ted in this c hapter, but that is b ey ond the scop e of this thesis. The size of the equations ab o v e and the dimensionalit y of the space of distributions mak e the original optimization problem in tractable, ho w ev er Guo and Rao (2008) dev elop ed a relaxation that admits a simple, closed form maxim um v alue b y emplo ying information ab out the constrain ts on the sym b ols imp osed b y virtue of the fact the q (k) (i1;j1):::(i ` ;j ` ) ’s describ e probabilities of outcomes of our sequen tial testing pro cedure. W e b egin b y partitioning the sums, separating o terms in whic h only one true n ull w as falsely rejected, and further separating those terms in whic h the single falsely rejected n ull w as rejected at the least p ossible signicance lev el. A t this p oin t w e still restrict the q (k) (i1;j1):::(i ` ;j ` ) ’s to b e those in the image of the rejection pro cedure applied to all v alid join t distributions: (1.3.17) 1 = sup m1+1 X k=1 m0 X i=1 1 k q (k) (i;k) + m1+1 X k=2 m0 X i=1 k1 X j=1 1 k q (k) (i;j) + m0 X `=2 m X k=` X (k) ` 1 k q (k) (i1;j1):::(i ` ;j ` ) (1.3.18) s:t: 8im 0 ;Jm; J^(m1+1) X j=1 q (j) (i;j) + J^(m1+1) X j=1 m1+1 X k=j+1 q (k) (i;j) + J X j=1 m X k=j_1 k^m0 X `=2 X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) J : Here w e replicate Lemmas 3.3 and 4.1 from Guo and Rao (2008), adapting only the notation for our sequen tial setting. Lemma 1.3. Symb olic Enc o ding of Pr o c e dur al Limits on q ’s F or (i 1 ;j 1 );:::; (i ` ;j ` )2 (k) ` with 1km and 1`k^m 0 , take j (1) :::j (`) to b e an asc ending or dering of j 1 ;:::;j ` . Then either the fol lowing two c onditions ar e satise d: (1.3.19) k`m 1 and j (d) k` +d 81d` or (1.3.20) q (k) (i1;j1):::(i ` ;j ` ) = 0: T o see that violation of the rst condition w ould require q (k) (i1;j1):::(i ` ;j ` ) = 0, notice that k` is the n um b er of false n ull h yp otheses correctly rejected. That n um b er m ust of course b e no greater than the total 1.3. MAIN RESUL TS 33 n um b er of false n ull h yp otheses. F or the second condition, consider the least signican t rejection lev el at whic h the d th most signican t false rejection could ha v e o ccurred. T o do so, rst assume that the k` true rejections o ccurred at the k` highest signicance lev els. Then the false rejection in question m ust ha v e tripp ed the k`+d lev el, otherwise there w ould ha v e to b e a gap somewhere b et w een the k` andk`+d signicance lev el where a barrier w as not tripp ed. No w dene i = 8 > > < > > : 1 i ; 1im 1 + 1 m1 i(i1) ; m 1 + 2im ; whic h w e use in the follo wing lemma. Lemma 1.4. Bound Imp ose d by Pr o c e dur al Limits Given the r estrictions imp ose d by L emma 1.3, for any p ermutation (i 1 ;j 1 );:::; (i ` ;j ` )2 (k) ` admitting a nonzer o q (k) (i1;j1):::(i ` ;j ` ) we have ` X d=1 j d 1=k: W e omit this pro of and ask the reader to reference Lemma 4.1 of Guo and Rao (2008) b ecause it is iden tical to the original and do es not dep end on an y sequen tial or ev en probabilistic framing b ey ond those conditions in Lemma 1.3. 1.3.3.4. Pr o of of The or em 1.1. Pr oof. With the restrictions and b ounds giv en in the preceding lemmas established, w e ma y no w treat these sym b ols as just that, dropping the notion of them as functionals and taking the suprem um o v er all v alues satisfying the established conditions. W e no w redene x ik :=q (k) (i;k) to distinguish what will b ecome the only relev an t sym b ols from the other irrelev an t ones, and to further signify that w e are lea ving b ehind the probabilistic meaning of an y of these sym b ols, ha ving already enco ded ev erything necessary in to the equations. Next, w e relax our problem b y augmen ting our ob jectiv e function with these additional terms m X k=m1+2 m0 X i=1 k x ik := m X k=m1+2 m0 X i=1 m 1 k(k 1) q (k) (i;k) (in whic h the q (k) (i;k) ’s in question are arbitrary v ariables), and add the follo wing terms to the relev an t con- strain ts J X j=m1+2 x ij := J X j=m1+2 q (j) (i;j) : Whic h presen ts the follo wing relaxed optimization problem: 1.3. MAIN RESUL TS 34 (1.3.21) 2 := sup m X k=1 m0 X i=1 k x ik + m1+1 X k=2 m0 X i=1 k1 X j=1 1 k q (k) (i;j) + m0 X `=2 m X k=` X (k) ` 1 k q (k) (i1;j1):::(i ` ;j ` ) (1.3.22) s:t: 8im 0 ;Jm; J X k=1 x ik + J^(m1+1) X j=1 m1+1 X k=j+1 q (k) (i;j) + J X j=1 m X k=j_1 k^m0 X `=2 X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) J and Equations 1.3.19 or 1.3.20 hold. Here w e sho w that FDR 1 2 , and then pro ceed to b ound 2 : An y collection of v alues of the x’s and q ’s that satisfy the constrain ts in Equation 1.3.18 will also satisfy those in Equation 1.3.22 b ecause 8i m 0 ; j m 1 + 2; w e ma y set x ik = 0, so w e ha v e only enric hed the space of sym b ols satisfying the constrain ts. A dditionally , Equation 1.3.21 will b e greater than or equal to (1.3.16), as w e ha v e only added p ositiv e terms. No w tak e an y solution that maximizes Equation 1.3.21 with q (k) (i1;j1):::(i ` ;j ` ) > 0 for some`> 1; and create a new solution where x i d j d = x i d j d +q (k) (i1;j1):::(i ` ;j ` ) for all 1 d ` and q (k) (i1;j1):::(i ` ;j ` ) = 0. F or this to still satisfy the constrain ts w e m ust sho w that for eac h constrain t this up date cannot in v alidate the inequalit y . Giv en x’s and q ’s that satisfy Equation 1.3.22, w e will sho w that the x ’s and q ’s as dened ab o v e also satisfy the same constrain ts, and can only increase the ob jectiv e function in Equation 1.3.21. Construct a new constrain t (1.3.23) 8im 0 ;Jm; J X k=1 x ik + J^(m1+1) X j=1 m1+1 X k=j+1 q (k) (i;j) + J X j=1 m X k=j_1 k^m0 X `=2 X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) J : No w for xed i and J , consider the corresp onding constrain t in Equation 1.3.22. F or eac h x ik in Equation 1.3.23, the set of q (k) (i1;j1):::(i ` ;j ` ) ’s whose mass it absorbs are exactly those in [ m k=2_j [ k^m0 `=2 (k) ` (i;j), (k 2 b ecause ` 2, and the total n um b er of rejections m ust meet or exceed the total n um b er of false rejections) eac h of whic h app ear in Equation 1.3.22, so no mass is b eing transferred in to Equation 1.3.23 from outside its equiv alen t in Equation 1.3.22. F or eac h q (k) (i1;j1):::(i ` ;j ` ) in Equation 1.3.22, i only app ears in (i 1 ;j 1 ):::(i ` ;j ` ) once, and all of the x ’s in it m ust b e of the form x i , so the mass that lea v es q (k) (i1;j1):::(i ` ;j ` ) is only transferred in to at most one x ik in Equation 1.3.23. Giv en q (k) (i1;j1):::(i ` ;j ` ) , for all pairs (i 1 ;j 1 );:::; (i ` ;j ` ), w e m ust ha v e j d k , whic h implies that j d 1 k and th us 1.3. MAIN RESUL TS 35 j d x i d j d + 1 k q (k) (i1;j1):::(i ` ;j ` ) = j d x i d j d 1 k q (k) (i1;j1):::(i ` ;j ` ) : So these transfers of mass cannot decrease of the v alue of the ob jectiv e. Incorp orating these up dates w e ha v e: 2 = sup m X k=1 m0 X i=1 k x ik + m1+1 X k=2 m0 X i=1 k1 X j=1 1 k q (k) (i;j) s:t: 8im 0 ; Jm; J X k=1 x ik + J^(m1+1) X j=1 m1+1 X k=j+1 q (k) (i;j) J F or an y q (k) (i;j) > 0 set x ik = x ik +q (k) (i;j) and q (k) (i;j) = 0. Again, this do esn’t c hange feasibilit y and can only increase the ob jectiv e b ecause k 1=k . This simplies to 2 = sup m X k=1 m0 X i=1 k x ik = sup m0 X i=1 m X k=1 k x ik = m0 X i=1 sup m X k=1 k x ik s:t: 8im 0 ; Jm; J X k=1 x ik J with the last equalit y in the ob jectiv e justied b y the lac k of constrain ts in v olving terms with dieren t i v alues. Due to symmetry , this in turn simplies further to 2 = sup m 0 m X k=1 k x 1k s:t: 8Jm; J X k=1 x 1k J No w giv en that k are decreasing in k , maximization of this ob ject in v olv es greedily forcing as m uc h mass as p ossible in to the x 1k ’s with lo w est k -v alues, and the max v alue is D(m 0 ;m 1 ;~ ) whic h pro v es Theorem 1.1. 1.3.3.5. pFDR V ariant. W e presen t here the mo dications of the to ols used to pro v e Theorem 1.1 nec- essary to pro v e Theorem 1.3. The follo wing lemmas are similar to Lemmas 1.1 and 1.2, but adapted to pFDR and pFNR con trol. Note that in order to dev elop b ounds on pFDR w e m ust assume that w e kno w the n um b er of true n ulls, b ecause if m 0 = 0 then pFDR = 0, and if m 0 =m then pFDR = 1. Lemma 1.5. pFDR p Symb ol De c omp osition 1.3. MAIN RESUL TS 36 F or the nite horizon, r eje ctive step-down pr o c e dur e describ e d in A lgorithm 1, using cutos A j as in Equation 1.3.1 and (1.3.24) p ijk =P ( i ( i )2 [A j ;A j1 ); i <T; R =kjR> 0); wher e the rst m 0 nul l hyp otheses ar e true, and the r emaining m 1 =mm 0 ar e false, we may de c omp ose pFDR as (1.3.25) pFDR = m0 X i=1 m X j=1 m X k=j 1 k p ijk : Pr oof. pFDR =E [V=RjR> 0] = m X k=1 E [VjR =k] k P (R =kjR> 0) = m X k=1 m0 X i=1 P H i 0 rejected; R =kjR> 0 k = m0 X i=1 m X k=1 k X j=1 P H i 0 rejected at lev elj; R =kjR> 0 k = m0 X i=1 m X j=1 m X k=j 1 k p ijk : W e can no w frame a pFDR equiv alen t to Equation 1.3.11. (1.3.26) pFDR 1 := sup m0 X i=1 m X j=1 m X k=j 1 k p ijk s:t: 8im 0 ; Jm; P ( i <T; i ( i )A J ) = J X j=1 m X k=j p ijk P 0 (R> 0) J : Here, the p ijk sym b ols dep end on the distribution P2P o v er whic h the suprem um is b eing tak en. Ho w ev er, for pFDR b ounds require some kno wledge of the true probabilit y of at least one rejection, so w e include P 0 (R> 0) in place of P (R> 0). This do es not violate the constrain t b ound, as w e can see if w e consider g (fp ijk (P )g;P ) to b e P ( i < T; i ( i ) A J ) = P J j=1 P m k=j p ijk P 0 (R> 0) as ab o v e, where the second argumen t is used only in the P (R> 0) term, w e ha v e 1.3. MAIN RESUL TS 37 g (fp ijk (P 0 )g;P 0 ) sup P g (fp ijk (P )g;P 0 ): Lemma 1.6. p to q Symb ols for pFDR Contr ol F or the nite horizon, r eje ctive step-down pr o c e dur e describ e d in A lgorithm 1, using cutos A j as in Equation 1.3.1, p ijk as in Equation 1.3.25, (k) ` and (k) ` (i;j) as in Equations 1.3.13 and 1.3.14, and (1.3.27) q (k) (i1;j1):::(i ` ;j ` ) =P (R =k;8d` i d <T \ i d ( i d )2 [A j d ;A j d 1 ) 8i2f1;:::;m 0 gnfi 1 ;:::;i ` g i =TjR< 0); we have (1.3.28) p ijk = k^m0 X l=1 X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) : Pr oof. Again, q sym b ols in (k) ` (i;j) represen t disjoin t ev en ts, so their sum is equiv alen t to the proba- bilit y of the union of their ev en ts. Because (k) ` (i;j) is, b y denition, all of the sets ((i 1 ;j 1 );:::; (i ` ;j ` ))2 (k) ` that include the pair (i;j) w e ha v e X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) =P i ( i )2 [A j ;A j1 ); i <T; R =k; V =`jR> 0 : Summing o v er ` w e ha v e k^m0 X l=1 X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) = k^m0 X l=1 P i ( i )2 [A j ;A j1 ); i <T; R =k; V =`jR> 0 =P i ( i )2 [A j ;A j1 ); i <T; R =k; V =`jR> 0 =p ijk : This in turn leads to the next represen tation of the optimization, equiv alen t to Equation 1.3.16: (1.3.29) pFDR 2 = sup m0 X `=1 m X k=` X (k) ` ` k q (k) (i1;j1):::(i ` ;j ` ) 1.3. MAIN RESUL TS 38 s:t: 8im 0 ;Jm; J X j=1 m X k=j k^m0 X l=1 X (k) ` (i;j) q (k) (i1;j1):::(i ` ;j ` ) J =P 0 (R> 0): Pro of of Theorem 1.3 Pr oof. No w b ecause all of the q ’s in our optimization, Equations 1.3.17 and 1.3.18, use k> 0, Lemmas 1.3 and 1.4 apply directly . T o see this, note that Lemma 1.3 details whic h q ’s m ust b e 0, but do es not sa y an ything ab out the v alue of an y p ossibly non-zero q sym b ols. F rom that p oin t on, the pro of the Theorem 1.3 is iden tical to that of Theorem 1.1 except that instead of using ~ = ( i ) i2f1;:::;mg w e use ~ = ( i =P 0 (R> 0)) i2f1;:::;mg . By the linearit y of D , w e get pFDRD ( i =P 0 (R> 0)) i2f1;:::;mg =D ( i ) i2f1;:::;mg =P 0 (R> 0): 1.3.3.6. Innite Horizon T ests with A c c eptanc e and R eje ction. F or the innite horizon, acceptiv e-rejectiv e v arian t, w e decomp ose FNR and pFNR in an iden tical manner to FDR and pFDR, as in Lemmas 1.1 and 1.5. Clearly FDP and FNP will tend to b e correlated. Ho w ev er, giv en that w e only care ab out E [FDP] and E [FNP], or E [FDPjR> 0] and E [FNPjR 0 > 0], w e ma y decouple their analyses. The only signican t dierence b et w een the innite horizon (IH) and nite horizon (FH) decomp ositions b oil do wn to the stopping time b ounds (FH) p ijk = P i ( i )2 [A j ;A j1 ); i <T; R =k (IH) p ijk = P i ( i )2 [A j ;A j1 ); R =k : These are unnecessary in the innite horizon v arian t b ecause if the test statistic for stream i is in the in terv al [A j ;A j1 ) up on termination, it m ust ha v e b een rejected. The sym b ols in the t yp e 2 decomp ositions will tak e the forms (FNR) p 0 ijk = P i ( i )2 (B j1 ;B j ]; R 0 =k (pFNR) p 0 ijk = P i ( i )2 (B j1 ;B j ]; R 0 =kjR 0 > 0 : The suprem um is tak en o v er the space of distributions P = P :2 H 1 0 [H 1 1 H 2 0 [H 2 1 ::: (H m 0 [H m 1 ) ; 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 39 ensuring that for eac h h yp othesis pair, the true distribution satises exactly on of the h yp otheses. F rom there the pro ofs of FDR and FNR con trol in Theorem 1.2 are iden tical to that of FDR con trol in Theorem 1.1. Lik ewise, pFDR and pFNR con trol in Theorem 1.4 are eac h sho wn using the same pro of used to sho w pFDR con trol in Theorem1.3. 1.4. Sim ulation Studies and Data Analysis In order to illustrate the actual p erformance of the metho ds describ ed in this thesis, w e dev elop ed a Python pac k age to fully implemen t all of the sp ecied testing pro cedures and run a v ariet y of Mon te Carlo (MC) sim ulations to ev aluate their op erating c haracteristics. See Chapter 3 for a complete description of the Python pac k age. W e presen t the results of these sim ulations using b oth syn thetic data in Section 1.4.1 and real drug side eect data in Section 1.4.2. W e use the syn thetic data sim ulations to compare sample size sa vings in Section 1.4.1.3, demonstrating the ecacy of the prop osed metho ds. 1.4.1. Sim ulated Data. In this section w e presen t the results of sim ulation studies of the sequen tial step-do wn pro cedures describ ed in Algorithms 1 and 2 in order to ev aluate their op erating c haracteristics and compare them to analogous xed-sample pro cedures. In the rst subsection, w e will describ e the distributional set ups for our sim ulations. In its sequels w e presen t the results of t w o metho ds of comparison b et w een sequen tial and xed-sample pro cedures. 1.4.1.1. Data Gener ation. W e explore p erformance on a v ariet y of dieren t distributional setups, detailed at the end of this subsection. In general, w e tak e the rst m 0 h yp otheses to b e the true n ulls, and the remaining m 1 to b e the alternativ es. This is relev an t b ecause of the correlation structure, discussed b elo w. F or eac h MC sim ulation, regardless of pro cedure or distribution, w e sim ulated all of the data ahead of time b ecause of limitations in an older v ersion of our soft w are pac k age, ho w ev er this results in a quic k er run time p er sim ulation, as the data can b e generated in a single v ectorized op eration. The most recen t v ersion of the pac k age allo ws for truly streaming, on the y data generation. In the nite horizon case this amoun ts to just sim ulating mT data p oin ts. Ho w ev er, in the innite horizon case w e m ust ensure that w e generate data for enough time steps to ensure, with some reasonable lev el of probabilit y , that all streams terminate b efore w e run out of data. W e c ho ose this horizon ~ T via an ad ho c but eectiv e approac h inspired b y the A v erage Sample Num b er, abbreviated as ASN, calculations for single stream sequen tial testing outlined in Go vindara julu (1975); further details are pro vided in Section A.1.3. In order to examine the p erformance of our pro cedures under correlated data, w e generate X i t using in v erse CDF’s of data generated from a Gaussian copula (T riv edi et al., 2007) as follo ws. F or eac h time step t = 1; 2;:::;T , in the nite horizon case, or t = 1; 2;:::; ~ T , in the innite horizon case, w e rst generate an 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 40 m dimensional sample of a Gaussian U t N(0; ) random v ariable, where (1.4.1) ij = jijj 8i;j2f1;:::;mg , then dene V i t = 1 (U i t ) and nally c ho ose X i t = max n2N 0 :P (XnjX P oisson ( i ))<V i t for the P oisson cases or X i t = 1 V i t <p i for the Binomial case, for eac h i2f1;:::;mg. This allo ws us to lo osely sp ecify a correlation structure while main taining the sp ecied marginal distributions. It also allo ws for simple violation of the PRDS condition, discussed in greater detail in Chapter 2, b y setting < 0. Finally , w e use this data to assem ble mT (or m ~ T ) grids of LLR statistics, as describ ed in Section 1.2.0.2. Once w e ha v e c hosen the st yle of the cutos, w e translate that in to the relev an t A i andB i v ectors using either the m ultiple W ald appro ximations describ ed in Section A.1.2 for innite horizon, acceptiv e-rejectiv e tests or the MC imp ortance sampling approac hes describ ed in Section A.1.3 for nite horizon rejectiv e pro cedures. F or the imp ortance sampling, w e used B reps = 1 100 max im1 1 i+1 i MC sim ulations, with an imp ortance sampling distribution shifted b ey ond the alternativ e h yp othesis, de- scrib ed in Section A.1.3. Using these cutos, w e ran eac h case and sub case 100 times, unless otherwise noted. Case 1. W e examine X i t P oisson( i ) data with =0:6, m = 10, 0 = 1:5, 1 = 2:0, = :25, and =:15. W e use BL st yle cutos, scaled using Guo-Rao factor (Equation 1.3.5) unless otherwise noted (as in Figure 1.4.1). The nite horizon uses T = 50. W e sim ulate three dieren t sub cases, where (m 0 ;m 1 ) = (3; 7), (5; 5), and (7; 3), and apply b oth the innite horizon, acceptiv e-rejectiv e Algorithm 2 and the nite horizon rejectiv e Algorithm 1. W e sim ulated eac h of these cases 1000 times. Case 2. Iden tical to Case 1 except that w e generate X i t Binomial (1;p) data with p 0 = 0:05 and p 1 = 0:15. Again, w e sim ulated these cases 1000 times. Case 3. Similar to Case 1 except that m = 100, m 0 = m 1 = 50, = :25, = :15, and T = 250. W e examine b oth Guo-Rao scaled cutos, and unscaled cutos. Case 4. W e examine X i t P oisson( i ) data with m = 100, 0 = 1:5, 1 = 2:0, = :25, = :15, and T = 250; w e use BL st yle cutos, scaled using Guo-Rao factor. Ho w ev er, in this case the data is not sim ulated from either the n ull or alternativ es, but rather i = exp log ~ 0 + i m log ~ 1 log ~ 0 where ~ 0 = 9 8 < 1:5 and ~ 1 = 8 3 > 2:0, so w e ha v e a grid of data generation sc hemes. Again, w e 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 41 apply b oth the innite horizon, acceptiv e-rejectiv e Algorithm 2 and the nite horizon rejectiv e Algorithm 1. The results of these sim ulations app ear in Figures 1.4.1 through 1.4.4. In the ma jorit y of these plots, the horizon tal axis will represen t rejection rates under the nite horizon rejectiv e pro cedure, whereas the v ertical axis will represen t rejection rates under the innite horizon, acceptiv e-rejectiv e pro cedure. F or eac h h yp othesis, there will b e a smaller, more transparen t glyphs, colored according the truth ab out its n ull h yp othesis, as describ ed in the caption of Figure 1.4.2; additionally , for eac h relev an t cluster, w e presen t a larger, less transparen t cen troid glyph. The correlation structure describ ed ab o v e will cause dieren t h yp otheses to p erform dieren tly , ev en if they are b oth n ull or b oth alternativ e, th us it is w orth while to presen t the individual h yp othesis glyphs instead of just the cen troids. Figures 1.4.1 and 1.4.2 con tain plots con trasting the p erformance of the pro cedures when the cutos are and aren’t scaled using the Guo-Rao factor, w e presen t the results of the scaled pro cedures using triangular glyphs, and those of the unscaled using circular glyphs. 1.4.1.2. T esting Pr o c e dur es and Observe d Op er ating Char acteristics. Here w e presen t further insigh ts in to the op erating c haracteristics of the algorithms describ ed in this c hapter b y applying them to the data generating cases laid out in the previous section. W e use BL st yle cuto v ectors as in Equation 1.2.7, with nominal v alues c hosen after considering the total n um b er of h yp otheses, as the BL st yle cutos saturate when 1=m, and examine the p erformance when the cuto v ector is left as dened b y the BL equations, as w ell as when it is scaled b y the Guo-Rao factor to accommo date dep endence. Error rates from b oth P oisson (Case 1) and Binomial (Case 2) sim ulations are presen ted in T able 3. Giv en that the ac hiev ed FDR, and FNR in the innite horizon setting, is w ell b elo w the nominal con trol rate, w e nd that ev en with a relativ ely pathological data generating sc heme, the b ounds tend to b e conserv ativ e. Ho w ev er ev en with the conserv ativ e b ounds, the pFDR and pFNR outcomes indicate that the practitioner should b e sk eptical of certain decisions reac hed b y the pro cedure. F or instance, in the nite horizon v arian t with P oisson data and m 0 = 3, if at least one n ull h yp othesis w as accepted, the probabilit y that it w as falsely accepted is on a v erage 60:74%. 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 42 P oisson (3, 7) (5, 5) (7, 3) Innite Horizon FDR 0.0252 0.0290 0.0390 FNR 0.0255 0.0186 0.0171 pFDR 0.1353 0.1894 0.2803 pFNR 0.2747 0.1876 0.1364 Rejectiv e FDR 0.0006 0.0017 0.0030 FNR 0.6074 0.4011 0.2234 pFDR 0.2917 0.4167 0.6000 pFNR 0.6074 0.4011 0.2245 Binomial (3, 7) (5, 5) (7, 3) Innite Horizon FDR 0.0223 0.0231 0.0344 FNR 0.0299 0.0214 0.0181 pFDR 0.1431 0.1775 0.2667 pFNR 0.2740 0.1798 0.1351 Rejectiv e FDR 0.0008 0.0020 0.0047 FNR 0.5862 0.3867 0.2187 pFDR 0.2611 0.3333 0.5185 pFNR 0.5874 0.3894 0.2275 T able 3. A c hiev ed error rates for Cases 1 and 2. Under indep endence w e w ould exp ect the sim ulations with unscaled cutos to satisfy the error con trol b ounds, ho w ev er our correlated data causes a breakdo wn in those guaran tees. In Figures 1.4.1 and 1.4.2 w e compare the rejection rates of n ull, alternativ e, and indeterminate h yp otheses under the nite horizon rejectiv e and innite horizon sc hemes, with b oth the scaled and unscaled v alues. F or the true n ulls, w e see a small degree of ination in false rejections when the v alues aren’t scaled under b oth sc hemes, as evidenced b y the blue triangles (represen ting the false rejection rate of the true n ulls when unscaled) b eing further to the righ t (higher rejection rate under nite horizon) and ab o v e (higher rejection rate under innite horizon) the blue dots (represen ting the false rejection rates when scaled). This is as w e w ould exp ect: requiring a more stringen t threshold for rejection results in few er false rejections. F or the alternativ es, under the nite horizon pro cedure w e notice the exp ected increase in true rejections, evidenced b y the red triangles shifting righ t w ard from the red circles. Ho w ev er, under the innite horizon pro cedure, the lac k of scaling applied to b oth the acceptance and rejection cutos actually reduces the rate of true rejections, evidenced b y the red triangles shifting sligh tly do wn w ard from the red circles. The discrepan t eects of scaling on true rejection rates of the t w o v arian ts arises b ecause the p o w er of the nite horizon v arian t is determined en tirely b y the rejection cutos and the horizon time, but the p o w er of the innite horizon v arian t is determined b y b oth the FDR con trolling A v ector, and the FNR con trolling B v ector, b oth of whic h are sub ject to scaling. In the unscaled exp erimen ts, w e k eep the horizon time the same but cease to scale bac k the rejectiv e cutos, allo wing for more rejections, b oth true and false. In Case 4, w e generate data for eac h h yp othesis from a range of rates, including some b elo w, b et w een, and ab o v e the n ull and alternativ e v alues. The rejection rates are sho wn in the righ t panel of Figure 1.4.2, where blue p oin ts are those b elo w the n ull, green are in b et w een the n ull and alternativ e, and red p oin ts are ab o v e the alternativ e. If w e pro ject the results on to the y-axis, sho wing the rejection rates in the innite horizon, general pro cedure, w e w ould see a tigh t red cluster of correctly rejected h yp otheses v ery close to 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 43 Figure 1.4.1. FDR con trolled and unscaled P oisson rejection rates as in Case 1 with (m 0 ;m 1 ) = (3; 7), (5; 5), and (7; 3), from left to righ t. Figure 1.4.2. Comparison of FDR rates in Cases 3 and 4. Blue p oin ts are true n ulls, red are false n ulls, and green p oin ts are the results of data generated from parameters that fall in b et w een the n ull and alternativ es. The larger glyphs are the cen troids of their resp ectiv e colors. 1, a tigh t blue cluster of correctly accepted h yp otheses v ery near 0, and a span of green p oin ts represen ting am biguous h yp otheses spanning the range b et w een 0 and 1, with a n um b er of them bunc hing up around 1, indicating questionable rejections. Pro jecting on to the x-axis, w e examine the rejection rate of the nite horizon rejectiv e pro cedure. Again, w e see the blue cluster near 0 and the green span from 0 to just short of 1, with a cluster v ery close to 0, indicating questionable acceptances. Ho w ev er the red cluster is m ust more spread out, due to the lac k of t yp e 2 error (FNR in this case) con trol in the nite horizon pro cedure. 1.4.1.3. Comp arisons to Fixe d-Sample Equivalents. One of the primary adv an tages of sequen tial testing is the p ossibilit y of a v oiding the p erils of mis-estimation of the necessary sample size, explored in depth in Bartro and Lai (2008). Bartro et al. (2012) sho ws that sequen tial pro cedures a v oid issues with loss of p o w er due to small samples, but w e demonstrate here that they also deliv er sa vings when the sample size is large. T o compare our pro cedures to the xed-sample equiv alen ts, w e tak e t w o approac hes, b oth using Cases 1 and 2. 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 44 The rst approac h to comparing sample sizes con trols nominal FDR at the same rate in b oth the sequen tial innite horizon, acceptiv e-rejectiv e pro cedure and the xed-sample pro cedure, and estimates the xed-sample size required to ac hiev e the same ac hiev ed FNR as the sequen tial pro cedure. W e presen t plots of ac hiev ed FDR and ac hiev ed FNR with resp ect to sample size for the xed-sample routines in Figures 1.4.3 and 1.4.4. In b oth cases the ac hiev ed FDR is w ell b elo w the nominal con trol lev el of = :25, ho w ev er it remains relativ ely constan t. The FNR app ears to deca y almost exp onen tially with sample size. W e use this observ ation to estimate the deca y rate of the log FNR of the xed-sample pro cedure using a linear regression mo del log FNR fixed = ^ c 0 + ^ c 1 T samples of log FNR on sample size, then in v ert that estimated mo del ^ T equiv = (log FNR seq ^ c 0 )=^ c 1 to come up with a range of candidate sample sizes around ^ T equiv for ac hieving the same FNR as our sequen tial pro cedure. F or further accuracy in our sample size matc hing, w e then test on that ner grid of xed-sample sizes, c ho osing the sample size that most precisely matc hes the desired FNR : The data presen ted in Figures 1.4.3 and 1.4.4 displa ys the op erating c haracterisics of the xed-sample pro cedures at all of the p oin ts tested in our searc h for the equiv alen t sample size, using the com bination of b oth the coarse and ne grids. FDR remains relativ ely constan t except in the case where m 0 = 3, b ecause the relativ ely few true n ull h yp otheses cause high v ariance at lo w sample sizes. 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 45 Figure 1.4.3. P oisson xed-sample error rates as compared to Case 1. The particularly noisy segmen ts w ere o v ersampled during the searc h for the xed-sample size whose ac hiev ed FNR most closely matc hed that of the relev an t sequen tial sim ulations. Figure 1.4.4. Binomial xed-sample error rates as compared to Case 2 . Again, the par- ticularly noisy segmen ts w ere o v ersampled during the searc h for the xed-sample size whose ac hiev ed FNR most closely matc hed that of the relev an t sequen tial sim ulations. The results are presen ted in T ables 4 and 5. In the upp er p ortion w e presen t the a v erage sample n um b er (ASN) for the sequen tial tests, splitting the streams in to groups based on their n ull and alternativ e status, as w ell as the nominal and ac hiev ed FDR and FNR. The section presen ts the results of a xed-sample equiv alen t with the sample size that most closely matc hed the FNR ac hiev ed b y the xed-sample pro cedure 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 46 to the ac hiev ed FNR of the sequen tial pro cedure. In the tables, this equiv alen t sample size is lab eled Equiv Sample Size, and sho ws sa vings of nearly 50%. Sequen tial (3, 7) (5, 5) (7, 3) Sequen tial (3, 7) (5, 5) (7, 3) H 0 ASN 55.9943 54.6568 50.4516 H 1 ASN 41.7769 46.0922 47.7147 Nominal FDR 0.250 0.250 0.250 A c hiev ed FDR 0.0242 0.0270 0.0384 Nominal FNR 0.150 0.150 0.150 A c hiev ed FNR 0.0217 0.0160 0.0172 Fixed-Sample (matc hing ac hiev ed FNR) Equiv Sample Size 98 103 95 Equiv FDR 0.0575 0.0376 0.0379 Equiv FNR 0.0233 0.0168 0.0172 T able 4. P oisson Comparison of equiv alen t sample size xed-sample tests to innite horizon sequen tial tests. See text for in depth discussion. Column headers are (m 0; m 1 ), the n um b er of true and false n ulls, resp ectiv ely . Sequen tial (3, 7) (5, 5) (7, 3) Sequen tial (3, 7) (5, 5) (7, 3) H 0 ASN 72.3713 72.1560 67.7357 H 1 ASN 46.0747 50.3256 52.6303 Nominal FDR 0.250 0.250 0.250 A c hiev ed FDR 0.0234 0.0262 0.0371 Nominal FNR 0.150 0.150 0.150 A c hiev ed FNR 0.0294 0.0198 0.0158 Fixed-Sample (matc hing ac hiev ed FNR) Equiv Sample Size 86 88 90 Equiv FDR 0.0593 0.0350 0.0289 Equiv FNR 0.0266 0.0190 0.0162 T able 5. Binomial Comparison of equiv alen t sample size xed-sample tests to innite hori- zon sequen tial tests. See text for in depth discussion. Column headers are (m 0; m 1 ), the n um b er of true and false n ulls, resp ectiv ely . Our second metho d of sample size estimation compares the nite horizon rejectiv e pro cedure with the xed-sample pro cedure with the same nominal FDR con trol rate and whose sample size matc hes that of the xed horizon. W e then compare total sa vings in samples from the alternativ e h yp otheses. The true n ull h yp otheses should not b e exp ected to stop b efore the horizon, and therefore will not manifest in sample size sa vings. These results are displa y ed in T able 6. W e see signican t sa vings in terms of sample size in all scenarios, with sup erior ac hiev ed FDR and comparable ac hiev ed FNR. Clearly some FNR p erformance is sacriced, as evidenced b y comparing the xed-sample ac hiev ed FNR ro w with the sequen tial ac hiev ed FNR ro w, but as the ratio of n ull h yp otheses to alternativ e h yp otheses increases, the dierence b ecomes marginal; notice the similarit y in FNR b et w een the pro cedures in b oth cases where m 0 = 7. In the sequen tial pro cedure, it is in teresting to note that the ratio of ac hiev ed FDR to nominal FDR app ears to b e relativ ely constan t as the nominal rate is v aried, as long as the h yp otheses remain unc hanged. This ratio c hanges as the total n um b er or mix of true and false h yp otheses v aries. 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 47 P oisson (3, 7) (5, 5) (7, 3) H 0 ASN 49.6470 49.7380 49.7487 H 1 ASN 36.3533 36.9400 37.3047 Seq A c hiev ed FDR 0.0109 0.0166 0.0312 Seq A c hiev ed FNR 0.4296 0.2634 0.1414 FS A c hiev ed FDR 0.0639 0.0345 0.0573 FS A c hiev ed FNR 0.1865 0.1627 0.1112 Binomial (3, 7) (5, 5) (7, 3) H 0 ASN 49.4180 49.7144 49.8193 H 1 ASN 36.2607 38.0390 39.2690 Seq A c hiev ed FDR 0.0158 0.0160 0.0211 Seq A c hiev ed FNR 0.3752 0.2591 0.1596 FS A c hiev ed FDR 0.0483 0.0271 0.0260 FS A c hiev ed FNR 0.2538 0.2023 0.1155 T able 6. Cases 1 and 2 nite horizon rejectiv e pro cedure results, compared to xed-sample pro cedures with sample size equal to the horizon of 50. Column headers are (m 0; m 1 ), the n um b er of true and false n ulls, resp ectiv ely . 1.4.2. Analysis of UK Y ello w card Pharmaco vigilence Database. In this section w e presen t the results of the application of our pro cedures to streaming rep orts of drug side eects in order to detect those drugs that cause a particular t yp e of side eect, so that they migh t b e remo v ed from mark et or otherwise sub jected to more restricted applications. As in Gur (2012), w e selected amnesia related side eects to b e the side eect of in terest in the application of our pro cedures. In order to collect the data necessary to base these sim ulations on real drugs, w e used a Python script to cra wl the PDF side eect summary rep orts for eac h of the appro ximately 2800 drugs in the UK Y ello w card Pharmaco vigilence Database. The Y ello w card data collection sc heme b egan in 1964, spurred b y the thalidomide crisis; its p opularit y has gro wn o v er the decades, and the MHRA recen tly released a mobile phone app to allo w the public to easily access the database (Wikip edia, 2017). F or eac h drug w e scrap ed the PDF rep orts from the In teractiv e Drug Analysis Proles section of the Y ello w card w ebsite on F ebruary 17, 2016, collecting the total n um b er of side eects, the n um b er of amnesia side eects, the starting date for collection of rep orts for the drug, and the date of the closure of the drug’s most recen t summary rep ort. The distribution of these data p oin ts as scrap ed from the summary rep orts can b e seen in Figure 1.4.5. Some of the drugs ha v e man y thousands of total side eect rep orts, collected o v er 7 decades. Ho w ev er, the ma jorit y of drug en tries ha v e v ery few rep orts, collected o v er less than a decade. 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 48 Figure 1.4.5. Heatmap of total Y ello w card side eect rep orts b y drug in the p erio d 1964 through 2016. There are quite a few drugs with v ery few total side eect rep orts, for instance Mammalian Blo o d with a total of t w o side eect rep orts. Including these drugs in our sim ulation w ould require running them for unreasonable lengths of time. Optimally , w e w ould lik e ha v e obtained the actual dates of eac h of the individual side eect rep orts, as w ell as the o v erall prescription rates of eac h drug. This w ould ha v e allo w ed us to examine our the metho dologies p erformance had it actually b een emplo y ed b y the UK health system, and to compare the rate of amnesia side eects to the o v erall p opularit y of the drugs. Unfortunately , there do es not seem to exist an y cen tral publicly a v ailable database of suc h prescription rates, if they are ev en computed at all; additionally , the MHRA informed us that collecting the full rep ort histories for eac h drug w as a more dicult request than they w ere able to accommo date giv en our purp oses. Instead, w e used the a v erage n um b er of total side eects rep orts p er y ear and a v erage n um b er of amnesia side eect rep orts p er y ear for eac h drug to sim ulate b oth amnesia and non-amnesia rep orts. W e then use the total n um b er of side eect rep orts as a pro xy for the p opularit y of eac h drug, and lo ok for drugs demonstrating a higher prop ortion of amnesia side eects, giv en their p opularit y . This sim ulation sc heme is similar to the parametric b o otstrap (Efron and Tibshirani, 1994), b y assuming that emissions of b oth amnesia and non-amnesia side eect rep orts o ccur as indep enden t P oisson pro cesses with rate parameters 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 49 drugi amnesia = (jfamnesia rep orts from drug igj + 1)= (rep ort i end date rep ort i start date) (1.4.2) drugi other = (jfnon-amnesia rep orts from drug igj + 1)= (rep ort i end date rep ort i start date): (1.4.3) These rates are visualized in Figure 1.4.6. The colorings corresp ond to the truth ab out their h yp othesis status, as discussed in the follo wing three paragraphs. Details of the data generation sc heme and LLR computation can b e found in Section A.1.3, sp ecically Equation A.1.1. Figure 1.4.6. Drug side eect rates as calculated via Equations 1.4.2 and 1.4.3 from the Y ello w card database data. Blue p oin ts ha v e pp 0 , green ha v e p 0 <p<p 1 , and red p oin ts ha v e pp 1 , as dened b y Equations 1.4.4 and 1.4.5. W e screened and remo v ed from consideration an y drugs with few er than 5 amnesia side eect rep orts and those with few er than 50 total side eect rep orts, as the database con tains a large n um b er of v ery obscure and an tiquated drugs, for instance Mammalian Blo o d or W o ol F at, with to o little data to b e usefully examined here; again reference Figure 1.4.5, noticing the dense cluster of drugs on the b ottom left that ha v e 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 50 only b een on the mark et for a few y ears, with on the order of 10 total side eect rep orts. A dditionally , w e add one amnesia rep ort and one non-amnesia rep ort to eac h drug in the calculation of its rates in Equations 1.4.2 and 1.4.3 to further reduce issues in tro duced b y rare, exotic drugs, lik e higher v ariance and unreasonably large termination times. W e test the follo wing h yp othesis pairs regarding the p ortion of side eects from eac h drug that w ere amnesia-related: H i 0 :p i =p 0 H i 1 :p i =p 1 >p 0 where p 0 and p 1 are the 45th and 85th p ercen tiles of i amnesia = i amnesia + i other i=1;:::;m drugs . That is, p 0 := 50 th p ercen tile of i amnesia = i amnesia + i other (1.4.4) p 1 := 90 th p ercen tile of i amnesia = i amnesia + i other . (1.4.5) A t eac h time step w e generate a P oisson n um b er of amnesia and non-amnesia rep orts for eac h drug, whic h is equiv alen t to generating a P oisson n um b er of total rep orts for eac h drug and then taking the n um b er of amnesia rep orts to b e Binomially distributed with p = jfamnesia rep orts from drug igj jfall rep orts from drug igj : F or our sim ulation, w e use the data from all of the drugs in the database to calculate p 0 and p 1 , but only apply it to h yp othesis testing on the 50% of drugs with the highest rate of side eect rep orts p er y ears recorded. W e sim ulated and analyzed the prop osed sequen tial pro cedures under a n um b er of dieren t congura- tions. F or eac h of the follo wing congurations, w e con trolled FDR at = 0:05 and FNR, in the innite horizon, acceptiv e-rejectiv e cases, at = 0:02. W e use = 0, i.e. indep enden t streams, b ecause correlating that man y streams w as computationally infeasible; generating join tly correlated data from appro ximately 1800 random v ariables in the manner describ ed in Section 1.4.1.1 w ould require in v erting a somewhat ill conditioned 1800 1800 matrix, whic h is an op eration of order O n 3 , and pro v ed to b e to o taxing for our Go ogle Cloud Compute instance to manage ev en once, m uc h less for eac h MC sim ulation. Because the data 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 51 is uncorrelated and relativ ely high dimensional, w e felt that tigh ter error con trols w ere required to elicit non-trivial outcomes. Case 1. BL cutos, as in Equation 1.2.7, using Algorithm 2 and T = 1000. Case 2. BL cutos using Algorithm 1 with T = 1000. Case 3. BL cutos using Algorithm 1 with T = 2000. Case 4. BL cutos using Algorithm 1 with T = 5000. Case 5. BH cutos, as in Equation 1.2.6, using Algorithm 2 and T = 1000. Case 6. BH cutos using Algorithm 1 with T = 1000. Case 7. BH cutos without Guo-Rao scaling using Algorithm 2 with T = 1000. Case 8. BH cutos without Guo-Rao scaling using Algorithm 1 with T = 1000. Figure 1.4.7. Stream (drug) terminations b y step for one MC sim ulation of the the se- quen tial step-do wn pro cedures using BL st yle cutos. The left panel presen ts Case 1. The righ t panel juxtap oses Cases 2 and 3, demonstrating the increase in rejections with a larger horizon when con trolled at the same FDR lev el. 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 52 Figure 1.4.8. Rejection rates for drugs in Case 1 and 2, where T = 1000. Figure 1.4.9. Rejection rates for drugs in Case 1 and 4, where T = 5000. 1.4. SIMULA TION STUDIES AND D A T A ANAL YSIS 53 Figure 1.4.10. Comparison of Cases 5 and 6 on the left to Cases 7 and 8 on the righ t, where the Guo-Rao scaling is not applied. Comparing the eect of the nite horizon b y examining Figures 1.4.8 and 1.4.9, w e notice a stark con trast. The nite horizon pro cedure p erforms p o orly , failing to detect a large n um b er of drugs that seem to cause amnesia when the horizon time is to o short, as demonstrated b y all of the red dots across the top of Figure 1.4.8, indicating that the innite horizon pro cedure rejected them regularly while the nite horizon pro cedure had mixed results. With a larger horizon, it tends to matc h the p erformance of the innite horizon pro cedure, as demonstrated b y the x-y symmetry of Figure 1.4.9. In Figure 1.4.10 w e examine p erformance with BH cutos, and compare the eect of the Guo-Rao scaling. W e nd that it has a signican t eect on FDR in the nite horizon case, as evidenced b y the increase in rejections depicted b y the righ t w ard shift of man y of the green and red dots. The scaling factor at this high of a dimension is signican t, so some impro v emen t should b e exp ected. In Case 1, out of all of the drugs detected to ha v e caused amnesia in more than 95% of MC sim ulations, the 10 drugs with the earliest a v erage stopping time are presen ted in T able 7, all of whic h are p opular drugs, kno wn to o ccasionally cause amnesia, as can b e conrmed via a quic k Go ogle searc h. 1.5. CONCLUSIONS 54 Drug Name Go ogle searc h hits with amnesia Go ogle searc h hits Ratio i A dalim umab 5880 174000 0.033793 1.096 Bupropion 16100 307000 0.052443 0.824 Clostridium T etani 206 52600 0.003916 1.000 Clozapine 10900 168000 0.064881 0.000 Etanercept 4770 146000 0.032671 1.000 Ethin ylo estradiol 1930 13600 0.141912 2.076 Rofeco xib 278 55400 0.005018 2.044 T elaprevir 617 62100 0.009936 1.712 V arenicline 6770 56400 0.120035 0.472 V enlafaxine 12000 205000 0.058537 1.472 T able 7. T op 10 drugs detected b y the sim ulations in Section 1.4.2 to ha v e caused amnesia, with the n um b er and ratio of Go ogle searc h hits for the drug name and the w ord amnesia to Go ogle searc h hits for the drug name alone. It is imp ortan t to note that, while these drugs ma y b e kno wn to cause amnesia, the results presen ted here are the outcomes of sim ulations based on summary statistics, not the real data. A dditionally , our use of total side eect rep orts as a pro xy for drug p opularit y if fraugh t with issues. 1.5. Conclusions In this c hapter, w e ha v e sho wn that the FDR and related error metrics for a sequen tial step-do wn test of m ultiple h yp otheses can b e b ounded without an y concern ab out the join t distribution of test statistics. W e did so b y adapting Guo and Rao’s (2008) optimization-based analysis of FDR-con trolling, xed-sample, step-do wn pro cedures to the sequen tial case, redening and extending the concepts and sym b ols they used and reducing it to the same nal upp er b ound. This b ound is usually v ery conserv ativ e, but is sho wn to b e eectiv e in a v ariet y of scenarios through sim ulation, ev en in the case of highly negativ ely correlated data streams, ac hieving error rates w ell b elo w the nominal con trol lev els. The exibilit y pro vided b y these disco v eries allo ws practitioners to emplo y these sequen tial testing pro ce- dures without ha ving to w orry ab out the estimating the dep endency structure of their data. Our sim ulations sho w that, while conserv ativ e, the sequen tial pro cedures allo w for signican t sa vings o v er xed-sample equiv- alen ts. This further reduces the burden on the practitioner, allo wing them to a v oid ha ving to estimate the optimal sample size a priori. CHAPTER 2 FDR Control With Positive Dependence 2.1. In tro duction and Motiv ation In this c hapter w e in tro duce a pair of sequen tial step-up pro cedures, dev elop ed b y Bartro and Song (2013), analogous to the sequen tial step-do wn pro cedures describ ed in Chapter 1, for sequen tial testing of m ultiple h yp otheses. W e pro ceed to adapt a theorem regarding FDR con trol in xed-sample pro cedures from Finner et al. (2009) to the sequen tial setting, and discuss sucien t mo dications of the nite horizon rejectiv e pro cedures presen ted in this c hapter and in Chapter 1 for application of our theorem. W e then apply our sequen tial step-up pro cedures, b oth with and without the mo dications, to sim ulated data, and compare them to xed-sample equiv alen ts as w ell as to the sequen tial step-do wn pro cedures discussed in this thesis, analyzing their op erating c haracteristics. In this c hapter w e fo cus en tirely on FDR con trol. W e remind the reader that if V is the n um b er of false rejections of true n ull h yp otheses, and R is the total n um b er of rejections of n ull h yp otheses, then w e dene the F alse Disco v ery Prop ortion (FDP) and F alse Disco v ery Rate (FDR) as (2.1.1) FDP :=V= (R_ 1) (2.1.2) FDR :=E[V=(R_ 1)] =E [FDP]: A ccording to Benjamini and Liu (1999), neither xed-sample step-up nor xed-sample step-do wn pro- cedures uniformly dominate the other in terms of p o w er when FDR con trol is desired. F or this reason, it is w orth examining b oth t yp es. As men tioned ab o v e, our sim ulations will explore b oth nite horizon, rejectiv e and innite horizon, acceptiv e-rejectiv e v arian ts of b oth sequen tial step-up and sequen tial step-do wn pro ce- dures. Our theoretical analysis fo cuses on b oth nite horizon, rejectiv e sequen tial step-do wn and step-up pro cedures, as dev elop ed b y Bartro (2014) for FDR con trol under a class of dep endency conditions inspired b y those p opularized b y Benjamini and Y ekutieli (2001), whic h in tro duced an analysis of FDR con trol of a xed-sample step-up pro cedure under Positive R e gr ession Dep endency on e ach one fr om a subset , or PRDS, 55 2.2. SEQUENTIAL STEP-UP PR OCEDURES 56 with the subset in question b eing those of the true n ull h yp otheses. This is essen tially a generalization of p ositiv e correlation to the m ultiv ariate case, and will b e explained in detail in Section 2.3. 2.2. Sequen tial Step-up Pro cedures In this section w e in tro duce t w o v arian ts of a sequen tial step-up pro cedure, analogous to those in Section 1.3.1. They are inspired b y and lev erage the xed-sample step-up pro cedure in tro duced b y Benjamini and Ho c h b erg (1995), whic h w e restate here in terms of arbitrary test statistics i for eac h h yp othesis i2f1;:::;mg, and cutos A j for j2f1;:::;mg sub ject to (2.2.1) 1 =A 0 A 1 :::A m : The xed-sample step-up pro cedure rejects the n ull h yp otheses with the k greatest order statistics where k = min j2f1;:::;mg : imj+1 A j ; and accepts the rest. Notationally , w e will emplo y the arg sort op erator in our descriptions of the sequen tial algorithms, whic h sorts the argumen ts or indices of an ordered set. F or instance, if X = (X 1 ;X 2 ;X 3 ) = (3; 5; 1) then (3; 2; 1) = (i 1 ;i 2 ;i 3 ) = arg sortX: 2.2.1. Sequen tial Finite Horizon, Rejectiv e Step-Up Pro cedure. Here w e in tro duce the sequen- tial nite horizon, rejectiv e step-up pro cedure dev elop ed b y Bartro and Song (2013). The pro cedure is v ery similar to its step-do wn equiv alen t, Algorithm 1 in Chapter 1. It tak es the same t yp e of input, m data streams, and m cutos A j sub ject to Equation 2.2.1 and the marginal conditions (2.2.2) 8im;2H i 0 P (9t<T s.t. i (t)A j ) j : It c hec ks for streams to b e terminated, along with their resp ectiv e cutos, and rejected at ev ery time step through its horizon time, at whic h p oin t it accepts the n ull h yp otheses of an y activ e streams. The dierence b et w een this algorithm and its step-do wn equiv alen t is in ho w it c ho oses whic h streams to terminate. A t eac h time step, a xed-sample step-up pro cedure, as discussed in Section 2.2, is p erformed on all remaining 2.2. SEQUENTIAL STEP-UP PR OCEDURES 57 statistics using all remaining un tripp ed cuto v alues; the n ull h yp otheses rejected b y the xed-sample pro- cedure are rejected, and the n ulls that it w ould ha v e accepted are returned for further testing. It’s formal statemen t is presen ted here. Algorithm 3 Finite Horizon, Rejectiv e Sequen tial Step-up Pro cedure 1: pro cedure ((X 1 ;:::;X m ); (A m ;:::;A 0 );T ) 2: k 0 . Num b er of rejections 3: t 0 4: I f1;:::;mg . Set of curren tly activ e h yp otheses 5: while t<T\k<m do . Collect data un til horizon is reac hed or all h yp otheses are rejected 6: t t + 1 7: i (t) i (X i 1:t ) 8i2I 8: i ? 1 ;i ? 2 ;:::;i ? mk arg sort i2I i (t) . See denition of arg sort 9: for d = 1; 2;:::;mk do 10: if i ? d (t)A md+1 then 11: I Infi ? d ;i ? d+1 ;:::;i ? mk g . Reject most signican t h yp otheses 12: k md 1 . Up date the n um b er of h yp otheses rejected. 13: Break F or Lo op 14: end if 15: end for 16: end while 17: A ccept I 18: Rejectf1;:::;mgnI 19: end pro cedure W e presen t an example demonstration of Algorithm 3 applied to Gaussian data with h yp otheses regarding the means in Figure 2.2.1. 2.2. SEQUENTIAL STEP-UP PR OCEDURES 58 Figure 2.2.1. An example of Algorithm 3 run on Gaussian data with (m 0 ;m 1 ) = (1; 4) and T = 20, with BH st yle cutos. The rst set of 3 sim ultaneous rejections are caused b y the third most signican t statistic crossing A 3 . 2.2.2. Sequen tial Innite Horizon, A cceptiv e-Rejectiv e Step-up Pro cedure. In this subsec- tion, w e presen t the sequen tial innite horizon, acceptiv e-rejectiv e step-up pro cedure, also dev elop ed b y Bartro and Song (2013). It mirrors the structure of the innite horizon step-do wn pro cedure describ ed in Algorithm 2 in Chapter 1; it requires the same input argumen ts with the same marginal guaran tees. Namely , m data streams; m rejection cutos A j sub ject to Equation 2.2.1 and the extra condition A m > 0; and m acceptance cutos (2.2.3) 1 =B 0 B 1 :::B m < 0: Both sets of cutos are sub ject to the marginal conditions 2.2. SEQUENTIAL STEP-UP PR OCEDURES 59 (2.2.4) 8im; jm;2H i 0 P (9t<1 s.t. i (t)A j \8t 0 <t (t 0 )>B 1 ) j (2.2.5) 8im; jm;2H i 1 P (9t<1 s.t. i (t)B j \8t 0 <t i (t 0 )<A 1 ) j : A t eac h time step, it c hec ks for streams to b e terminated, rst via acceptance then via rejection, along with their resp ectiv e cutos, pro ceeding in this fashion un til all h yp othesis pairs ha v e b een decided up on. As with Algorithm 3, at eac h time step it c ho oses h yp otheses for acceptance and rejection b y applying a xed-sample step-up pro cedure to the data and the most extreme remaining cutos; in the acceptance phase it accepts those selected b y the xed-sample pro cedure, and returns the rest for further testing. The rejection phase is trivially similar. As an example, consider at time step t there ha v e b een k a acceptances and k r rejections. Then the rejection thresholds passed to the xed-sample step-up pro cedure are the middle group out of the follo wing three groups: (1) A 1 ;:::;A k r are the most extreme. They ha v e already b een tripp ed, and are remo v ed from further consideration. (2) A k r +1 ;:::;A mk a are the most signican t y et-un tripp ed rejection cutos. Note that there are mk a k r of them, whic h is exactly the n um b er of activ e h yp otheses. (3) A mk a +1 ;:::;A m ha v e b een implicitly eliminated due to acceptances. If the i1 (t)<A mk a and i2 (t)<A mk a 1 , but i3 (t)A mk a 2 , w e reject and terminate h yp otheses i 3 through i mk a k r , and con tin ue testing h yp otheses i 1 and i 2 . It’s formal statemen t of the algorithm is presen ted here. 2.2. SEQUENTIAL STEP-UP PR OCEDURES 60 Algorithm 4 Innite Horizon, A cceptiv e-Rejectiv e Sequen tial Step-up Pro cedure 1: pro cedure ((X 1 ;:::;X m ); (A m ;:::;A 0 ); (B m ;:::;B 0 )) 2: k r 0, k a 0 . Num b er of rejections/acceptances. 3: t 0 4: I f1;:::mg . Set of curren tly activ e h yp otheses. 5: I r ,I a . Set of curren tly rejected/accepted h yp otheses. (Empt y) 6: while k r +k a <m do . Collect data un til all h yp otheses ha v e b een accepted or rejected. 7: t t + 1 8: i (t) i (X i 1:t ) 8i2I 9: i 1 ; i 2 ;:::; i mk a k r arg sort i2I i (t) 10: for d = 1; 2;:::;mk a k r do 11: if i d (t)B mk r d+1 then 12: I Inf i d ; i d+1 ;:::; i mk a k rg . A ccept most signican t h yp otheses 13: I a I a [f i d ; i d+1 ;:::; i mk a k rg 14: k a mk r d + 1 . Up date the n um b er of h yp otheses accepted. 15: Break F or Lo op 16: end if 17: end for 18: ^ i 1 ; ^ i 2 ;:::; ^ i mk a k r arg sort i2I i (t) 19: for d = 1; 2;:::;mk a k r do 20: if ^ i d (t)A mk a d+1 then 21: I Inf ^ i d ; ^ i d+1 ;:::; ^ i mk a k rg . Reject most signican t h yp otheses 22: I r I r [f ^ i d ; ^ i d+1 ;:::; ^ i mk a k rg 23: k r mk a d + 1 . Up date the n um b er of h yp otheses rejected. 24: Break F or Lo op 25: end if 26: end for 27: end while 28: Reject I r , A ccept I a 29: end pro cedure 2.3. FDR CONTR OL UNDER POSITIVE DEPENDENCY 61 Figure 2.2.2. An example of Algorithm 4 run on Gaussian data with (m 0 ;m 1 ) = (3; 2), with BH st yle cutos. 2.3. FDR Con trol Under P ositiv e Dep endency In this section w e in tro duce a theorem pro viding for FDR con trol of a class of nite horizon rejectiv e sequen tial m ultiple testing pro cedures under a p ositiv e dep endency condition. This w ork is motiv ated b y Benjamini and Y ekutieli’s (2001) theorem ab out FDR con trol in xed-sample step-up pro cedures under PRDS. Their pro of is complex and applies only to BH st yle cutos of the form (2.3.1) i =i=m 8i2f1;:::;mg for xed-sample step-up pro cedures. Ho w ev er, Finner et al. (2009) succeeded in simplifying the pro of and extending it to other cuto v ector structures, as w ell as other m ultiple rejection pro cedures. Here w e extend the Benjamini-Y ekutieli theorem, via an adaptation of Finner et al.’s pro of, to the sequen tial setting, applying it to b oth the nite horizon, rejectiv e sequen tial step-do wn pro cedures discussed in Chapter 1 and the nite 2.3. FDR CONTR OL UNDER POSITIVE DEPENDENCY 62 horizon, rejectiv e sequen tial step-up pro cedure presen ted in Algorithm 3 of this c hapter; though p oten tially feasible, w e do not extend our theorem to include arbitrary cuto structures, instead limiting it to BH st yle cutos, or the innite horizon, acceptiv e-rejectiv e v arian ts of the aforemen tioned rejections pro cedures. The primary mo dication necessary to extend Finner et al.’s w ork is the extension of the concept of PRDS to the sequen tial setting. T o explain PRDS in the xed-sample setting w e m ust rst in tro duce the concept of an increasing set: Definition 2.1. Increasing Set DR n is an increasing set if8x2D;y2R n w e ha v e xy elemen t-wise implies y2D . Using this denition, PRDS can b e dened as follo ws: Definition 2.2. PRDS on the the set I 0 F or all increasing sets D R m and8i2 I 0 f1;:::;mg, if P ~ 2Dj i =x is non-decreasing in x then w e sa y that P is PRDS on I 0 . F or m ultiple testing pro cedures, whether xed-sample or sequen tial, w e will b e concerned with ev en ts of the formfRkg where R is the n um b er of n ull h yp otheses rejected b y the pro cedure. F or a xed-sample pro cedure let ~ denote the v alues of the m test statistics (e.g. p-v alues). Since ~ uniquely determines the n um b er of rejections R, the ev en tfRkg is equiv alen t to n ~ 2D o for some set D: In other w ords (2.3.2) 8k2f0;:::;mg 9D2R m s.t. fRkg = n ~ 2D o : F or step-up and step-do wn tests, these sets D can b e sho wn to b e increasing sets, so if the true distribution of the data is PRDS on the set of n ull h yp otheses, then 8i2I 0 ;k2f1;:::;mg, P Rkj i =x is non- decreasing in x, and b y Lemma B.1 in Section B.1 P Rkj i x is also non-decreasing. The latter fact is crucial for Finner et al.’s pro of. The sequen tial setting requires a n um b er of mo dications and additional assumptions. T o in tro duce them w e use the notation of Chapter 1, with reminders where necessary , and the elemen t-wise (or stream-wise) maxim um op erator dened as elmax tT ~ (t) = max tT i (t) i2f1;:::;mg where8tT , ~ (t)2R m . Giv en that our marginal conditions in the sequen tial pro cedures are of the form 2.3. FDR CONTR OL UNDER POSITIVE DEPENDENCY 63 (2.3.3) sup 2H i 0 P max t<T i (t)A j j 8i;j2f1;:::;mg; w e w ould need8i2 I 0 ; k2f1;:::;mg, P Rkj max t<T i (t) =x non-decreasing in x. Here the j = j=m. When the pro cedure is suc h that 8k2f1;:::;mg, there exists some increasing set D2f1;:::;mg for whic h fRkg = n elmax tT ~ (t)2D o ; w e see that the PRDS condition on the stream maxim ums will b e sucien t for FDR con trol. Ho w ev er, the nite horizon pro cedures, as presen ted in the generalit y of Algorithm 1 in Chapter 1 and Algorithm 3 in this c hapter, will not necessarily satisfy this condition. W e state here the bare minim um conditions required for our pro of, then discuss metho ds for satisfying them. Condition 2.1. Requiremen ts On Pro cedures (1) A i v ector satisfying Equations 2.2.1 and 2.2.2 . (2) Equation 2.3.3 is satised with i =i=m, as in Equation 2.3.1. (3) Rejection of H i 0 giv en that Rk implies i ( i )A k for i;k2f1;:::;mg. The rst t w o items are easily addressed and reasonable assumptions to mak e ab out the parameters for a giv en pro cedure. The third limits the t yp es of pro cedures, stating that if n ull h yp othesis i w as rejected amongst at least k total rejections, then its statistic m ust ha v e exceeded the k th cuto at the time of its termination. Both our nite horizon pro cedures satisfy this requiremen t. Condition 2.2. Sucien t Condition (2.3.4) P Rk + 1 max tT i (t)A k P Rk + 1 max tT i (t)A k+1 0 for i2f1;:::;mg and k2f1;:::;m 1g. W e presen t the theorem and its pro of here, and discuss the implications of Condition 2.2 in the follo wing section. Theorem 2.1. FDR Contr ol of Se quential Finite Horizon R eje ctive Pr o c e dur es Under PRDS- like Conditions F or a se quential nite horizon, r eje ction pr o c e dur e with cuto ve ctor (A i ) i2f1;:::;mg and true, unknown joint distribution P satisfying the pr op erties in Conditions 2.1 and 2.2, FDR : 2.3. FDR CONTR OL UNDER POSITIVE DEPENDENCY 64 Pr oof. W e ha v e the follo wing c hain of equalities and inequalities, with individual steps explained b elo w. FDR = m0 X i=1 m X k=1 1 k P R =k; rejectH i 0 (2.3.5) m0 X i=1 m X k=1 1 k P R =k; i <T; i ( i )A k (2.3.6) = m0 X i=1 m X k=1 1 k P i <T; i ( i )A k jR =k P (R =k) (2.3.7) m0 X i=1 m X k=1 1 k P max t<T i (t)A k jR =k P (R =k) (2.3.8) = m0 X i=1 m X k=1 1 k P R =k; max t<T i (t)A k (2.3.9) = m0 X i=1 m X k=1 1 k P R =kj max t<T i (t)A k P max t<T i (t)A k (2.3.10) m0 X i=1 m X k=1 k k P R =kj max t<T i (t)A k (2.3.11) m0 X i=1 ( m1 X k=1 k k (P R =kj max t<T i (t)A k +P Rk + 1j max t<T i (t)A k (2.3.12) P Rk + 1j max t<T i (t)A k+1 ) + m m P Rmj max t<T i (t)A m ) (2.3.13) = m0 X i=1 ( m1 X k=1 k k (P Rkj max t<T i (t)A k P Rk + 1j max t<T i (t)A k+1 ) (2.3.14) + m m P Rmj max t<T i (t)A m ) (2.3.15) = m0 X i=1 m X k=1 k k P Rkj max t<T i (t)A k ! m X k=2 k1 k 1 P Rkj max t<T i (t)A k ! (2.3.16) = m0 X i=1 ( 1 P R 1j max t<T i (t)A 1 + (2.3.17) m X k=2 k k k1 k 1 P Rkj max t<T i (t)A k ) (2.3.18) = m0 X i=1 m 1 + m X k=2 0P Rkj max t<T i (t)A k ! (2.3.19) = m 0 m (2.3.20) 2.3. FDR CONTR OL UNDER POSITIVE DEPENDENCY 65 The rst inequalit y , is a result of Item 3 from Condition 2.1. The next inequalit y is a clear result of f ()Agfmax tT (t)Ag. The third inequalit y is clearly implied b y Item 2. Finally , Equation 2.3.12 tak es adv an tage of Condition 2.2. The follo wing line clev erly reorganizes the terms of the telescoping se- ries and tak es adv an tage of the fact that P Rmj max tT i (t)A m =P R =mj max tT i (t)A m . Equation 2.3.16 uses the fact that for BH st yle cutos k =k ==m. 2.3.1. Satisfying the Sucien t Conditions for Theorem 2.1. Condition 2.2, distilled from the implications of PRDS in the xed-sample setting, is the most complex and dicult condition to v erify . It requires kno wledge of b oth the join t distribution and the rejection pro cedure. The xed-sample theorem decouples these issues, with PRDS making a statemen t concerning only the distribution, and the nature of the pro cedure implying that Equation 2.3.2 is true with eac h D b eing an increasing set. One approac h to satisfying Condition 2.2 is to replace the sequen tial statistics with their historical maxim ums, substituting ~ i (t) := max t 0 t i (t 0 ) for i (t). In the nite horizon v arian ts of b oth algorithms discussed here, w e need not assume that the statistics are LLRs; aside from certain eciency guaran tee, the freedom to emplo y the W ald appro ximations are our main motiv ation for using them in the innite horizon v arian ts. Using the historical maxim um do es not violate an y of the assumptions; the marginal b ounds in Item 2 of Condition 2.1 still hold b ecause P max tT ~ i (t)A j =P max tT i (t)A j 8i;j2f1;:::;mg; 2 ; so no adaptations to the MC or imp ortance sampling sc hemes need b e in tro duced to accommo date the c hange. No w, for b oth of the nite horizon rejectiv e algorithms in tro duced here w e ha v e 8k2f1;:::;mg, there exists some increasing set D2f1;:::;mg suc h that fRkg = elmax tT ~ i (t) i2f1;:::;mg 2D = ~ i (T ) i2f1;:::;mg 2D ; whic h reduces to the n um b er of rejections in xed-sample equiv alen t pro cedure applied to ~ i (T ) i2f1;:::;mg . Therefore, if ~ i (T ) are PRDS on I 0 , Condition 2.2 is satised. In Figure 2.3.1, w e see a demonstration analogous to that in Figure 2.2.1, but with the historical maxim ums in place of the true LLR statistics. 2.4. SIMULA TIONS 66 Figure 2.3.1. An example of Algorithm 3 run on Gaussian data, as in Figure 2.2.1, but with the sequen tial LLR statistics replaced with their historical maxim ums. 2.4. Sim ulations Here w e will presen t the results of sim ulations similar to those in Chapter 1, though fo cusing on step- up pro cedures. W e start b y presen ting the general sim ulations t yp es, as in the step-do wn equiv alen t, then compare them against eac h other to explore their op erating c haracteristics and b eha vior under distributions conduciv e to satisfying Condition 2.2, and ones that violate it. In the follo wing subsection, w e examine the sa vings of the sequen tial step-up pro cedure o v er its xed-sample equiv alen t, as in Section 1.4.1.3. 2.4.1. Data Generation and Discussion. In addition to scenarios mimic king those in our step-do wn sim ulation sections, w e also in tro duce t w o more complex correlation structures. In the rst, referred to as SC for separate correlations, all of the n ulls are correlated with eac h other, as in Section 1.4.1.1, and all of the alternativ es are lik ewise join tly correlated, ho w ev er the n ull data streams are indep enden t of the alternativ e 2.4. SIMULA TIONS 67 data streams. The correlation matrix will th us ha v e the form = 0 B @ 0 0 0 1 1 C A where 0 and 1 are m 0 m 0 and m 1 m 1 matrices of the simpler form men tioned ab o v e. The second correlation structure uses the same st yle correlation matrix as in Equation 1.4.1 of Chapter 1, but for eac h MC sim ulation, it generates are random mm p erm utation matrix W and uses W W T as the correlation matrix for the Gaussian copula in that sim ulation, this increases the n um b er of n ull data streams closely correlated with alternativ e data streams. W e refer to this as R O. Our data generating and testing sc hemes are as follo ws. Case 1. W e examine X i t P oisson( i ) data with =0:6, m = 10, 0 = 1:5, 1 = 2:0, = :25, and = :15, w e use BH st yle cutos, scaled using log (m) factor unless otherwise noted. The nite horizon T = 50. W e sim ulate three dieren t sub cases, where (m 0 ;m 1 ) = (3; 7), (5; 5), and (7; 3), and apply the nite horizon rejectiv e Algorithm 3. W e sim ulated eac h of these cases 1000 times. Case 2. Iden tical to Case 1 except that w e generate X i t Binomial (1;p) data with p 0 = 0:05 and p 1 = 0:15. Again, w e sim ulated these cases 1000 times. Case 3. Iden tical to Cases 1 and 2, except w e replace i (t) with max t 0 t i (t 0 ), as discussed in Section 2.3.1. Case 4. Iden tical to Cases 1 and 2, except T = 75. Case 5. Iden tical to Case 3, using the correlation structure SC, discussed in the preceding paragraph, where b oth groups use = 0:6. Case 6. Iden tical to Case 3, except T = 75. Case 7. Iden tical to Cases 1 except that the streams are indep enden t. Case 8. Iden tical to Cases 1 except that the streams use the R O correlation structure. Figure 2.4.1. Comparison of P oisson sim ulations from Case 1 against Case 4, comparing the eect of a larger horizon. 2.4. SIMULA TIONS 68 In Figure 2.4.1, w e see notice that the horizon time T = 50 results in notably few er rejections, when its FDR scaling factor of logm scaling is applied, than the sequen tial step-do wn pro cedure with equiv alen t scaling under BL st yle cutos. W e th us sim ulated it with T = 75 and compared their results, noting that ev en under scaling, the larger horizon case correctly rejects the false n ulls the ma jorit y of the time. F urther details of these comparisons can b e seen in T ables 1 and 2. W e note that in the shorter horizon, FDR is lo w er and FNR is higher than in the larger horizon case; the sa vings in H 1 ASN are minimal. Figure 2.4.2. Comparison of P oisson rejection rates in Cases 7 and 8. In Figure 2.4.2 w e see that scaling app ears to ha v e little eect on the results when the data streams are indep enden t. Ho w ev er, when they are highly correlated, as in the R O setting, scaling drastically reduces p o w er while impro ving FDR con trol. 2.4. SIMULA TIONS 69 Figure 2.4.3. Comparison of the normal LLR statistics as in Cases 1 and 2 to the use of the historical maxim um, as in Case 3. Comparing the p erformance of otherwise iden tical pro cedures when the statistic is replaced with its historical maxim um, as describ ed in Section 2.3.1, w e nd that scaling has signican tly less impact on the outcomes when the historical max is used, indicating that satisfying Condition 2.2 drastically impro v es p o w er. W e also applied our metho ds to Y ello w card drug side eect data (MHRA, 2017), as in Section 1.4.2. W e compared the t w o nite horizon, rejectiv e pro cedures, b oth using unscaled BH st yle cutos, but otherwise sim ulated iden tically to the sim ulations in the aforemen tioned section. W e then rep eated the sim ulations and comparison, replacing the LLR statistics with the historical maxim um statistics. The results are presen ted in Figures 2.4.4 and 2.4.5. 2.4. SIMULA TIONS 70 Figure 2.4.4. Comparison of nite horizon rejectiv e step-up using BH st yle cutos with and without the historical max (CM for cum ulativ e max) on Y ello w card data. Figure 2.4.5. Comparison of nite horizon rejectiv e step-do wn using BH st yle cutos with and without the historical max (HM) on Y ello w card data. The most extreme div ersion from the x =y line in b oth comparisons is the drug Indomethicin. A ccording the an observ ational study of side eect rep orts collected b y the w ebsite eHealthMe (2017), 1.36% of side 2.4. SIMULA TIONS 71 eect rep orts for Indomethacin w ere amnesia rep orts, indicating that it do es indeed cause amnesia. The historical max, therefore, seems to p erform b etter than the classical LLR approac h for that drug. 2.4.2. Fixed-Sample Equiv alen ts. Again, w e rep eat the exp erimen ts in Section 1.4.1.3 comparing the p erformance and sa vings of nite horizon rejectiv e sequen tial pro cedures to their xed-sample equiv alen ts. P oisson P oisson Unscaled Binomial (3, 7) (5, 5) (7, 3) (3, 7) (5, 5) (7, 3) (3, 7) (5, 5) (7, 3) H 0 ASN 49.9680 49.9700 49.9687 47.7137 48.7166 49.2794 49.9547 49.9650 49.9674 H 1 ASN 43.3176 43.7030 44.1103 26.7760 31.3924 35.8420 44.1590 44.4132 44.4160 Seq FDR 0.0017 0.0031 0.0073 0.0374 0.0537 0.0680 0.0012 0.0029 0.0055 Seq FNR 0.5459 0.3600 0.2044 0.2341 0.1538 0.1078 0.5699 0.3835 0.2143 Seq FDR s.e. 0.0006 0.0010 0.0019 0.0021 0.0032 0.0045 0.0005 0.0010 0.0017 Seq FNR s.e. 0.0033 0.0027 0.0022 0.0060 0.0041 0.0030 0.0037 0.0030 0.0023 FS FDR 0.0402 0.0465 0.0508 0.0743 0.1316 0.1460 0.0386 0.0375 0.0757 FS FNR 0.1889 0.1097 0.0881 0.0767 0.0455 0.0367 0.2366 0.1884 0.0946 FS FDR s.e. 0.0069 0.0091 0.0131 0.0079 0.0116 0.0183 0.0068 0.0093 0.0141 FS FNR s.e. 0.0193 0.0129 0.0080 0.0148 0.0096 0.0069 0.0222 0.0135 0.0097 T able 1. Comparison of nite horizon, rejectiv e step-up pro cedure for xed-sample equiv- alen ts, as in Cases 1 and 2. P oisson P oisson Unscaled Binomial (3, 7) (5, 5) (7, 3) (3, 7) (5, 5) (7, 3) (3, 7) (5, 5) (7, 3) H 0 ASN 73.5440 73.7034 73.7819 72.1193 71.9598 72.4379 74.0767 74.3006 74.3750 H 1 ASN 38.0851 38.4446 38.7813 32.8729 33.6100 35.9467 42.5141 43.4294 43.7900 Seq FDR 0.0156 0.0304 0.0579 0.0275 0.0609 0.1060 0.0121 0.0189 0.0329 Seq FNR 0.2002 0.1076 0.0552 0.1240 0.0621 0.0348 0.2622 0.1562 0.0781 Seq FDR s.e. 0.0015 0.0024 0.0040 0.0018 0.0030 0.0049 0.0014 0.0019 0.0031 Seq FNR s.e. 0.0056 0.0037 0.0023 0.0051 0.0031 0.0021 0.0060 0.0039 0.0026 FS FDR 0.0230 0.0472 0.0683 0.0643 0.1181 0.1709 0.0329 0.0625 0.0795 FS FNR 0.0722 0.0410 0.0171 0.0200 0.0138 0.0088 0.0920 0.0855 0.0428 FS FDR s.e. 0.0055 0.0084 0.0130 0.0078 0.0115 0.0180 0.0065 0.0098 0.0166 FS FNR s.e. 0.0131 0.0079 0.0048 0.0084 0.0051 0.0036 0.0149 0.0115 0.0079 T able 2. Comparison of nite horizon, rejectiv e step-up pro cedure for xed-sample equiv- alen ts with T = 75, as in Case 4. Comparing the b olded ASN in T ables 1 and 2, w e notice that with a shorter horizon and scaled cutos, the sa vings are minimal; w e do see signican t impro v emen ts in FDR of the sequen tial pro cedures o v er the xed sample pro cedures, ho w ev er FNR suers more signcan tly in the sequen tial v arian ts. When the horizon is extended, w e note alternativ e h yp othesis sample size sa vings of nearly 50%, indicating that for sucien tly distan t horizons, sa vings will alw a ys tend to accrue. W e also notice less of a degredation in FNR in the sequen tial setting, as compared to the xed sample setting; this is unsurprising as, giv en xed t yp e 1 error con trol, t yp e 2 error is con trolled only b y the horizon time. W e further analyze the use of historical maxim um statistics, satisfying Condition 2.2, in T ables 3 and T ables 4and 5. The former directly compares the p erformance of the LLR approac h to the HM approac h 2.4. SIMULA TIONS 72 when run with iden tical settings. The latter t w o tables should b e compared against T ables 1 and 2. In the HM tables w e see sligh t decreases in ASN o v er the traditional LLR approac h, although this impro v emen t is not en tirely uniform in the T = 50 setting. The latter tables presen t the results of the most fa v orable setting a v ailable while still using correlated data, in terms of adherence to conditions men tioned b y Finner et al. (2009). This results in noticeable sa vings without exceeding FDR b ounds. (m 0 ;m 1 ) = (5; 5) P oisson P oisson Unscaled Binomial Normal Hist Max Hist Max Normal Normal Hist Max H 0 ASN 49.9700 49.6216 49.3452 48.7166 49.9650 49.8952 H 1 ASN 43.7030 37.1140 33.9824 31.3924 44.4132 40.7642 Seq FDR 0.0031 0.0232 0.0337 0.0537 0.0029 0.0070 Seq FNR 0.3600 0.2289 0.1974 0.1538 0.3835 0.3219 Seq FDR s.e. 0.0010 0.0023 0.0027 0.0032 0.0010 0.0014 Seq FNR s.e. 0.0027 0.0038 0.0040 0.0041 0.0030 0.0035 FS FDR 0.0499 0.0499 0.1282 0.1130 0.0535 0.0490 FS FNR 0.0898 0.1086 0.0505 0.0410 0.1714 0.1859 FS FDR s.e. 0.0089 0.0090 0.0108 0.0111 0.0104 0.0104 FS FNR s.e. 0.0109 0.0122 0.0097 0.0085 0.0130 0.0115 T able 3. Comparison of the normal LLR statistics as in Cases 1 and 2 to the use of the historical maxim um, as in Case 3. P oisson P oisson Unscaled Binomial (3, 7) (5, 5) (7, 3) (3, 7) (5, 5) (7, 3) (3, 7) (5, 5) (7, 3) H 0 ASN 49.9813 49.9818 49.9749 47.2140 48.1578 48.5399 49.9583 49.9650 49.9727 H 1 ASN 44.1516 44.5374 45.2347 28.3563 28.4436 32.1390 42.8660 43.6096 44.0727 Seq FDR 0.0016 0.0029 0.0031 0.0457 0.0665 0.1035 0.0021 0.0039 0.0054 Seq FNR 0.5402 0.3530 0.2075 0.2160 0.1443 0.0903 0.5209 0.3637 0.2076 Seq FDR s.e. 0.0007 0.0012 0.0011 0.0028 0.0040 0.0059 0.0006 0.0012 0.0019 Seq FNR s.e. 0.0053 0.0051 0.0037 0.0077 0.0055 0.0038 0.0057 0.0044 0.0031 FS FDR 0.0208 0.0405 0.1040 0.0616 0.1005 0.1855 0.0445 0.0416 0.0598 FS FNR 0.1431 0.1173 0.0819 0.0408 0.0639 0.0370 0.2361 0.1501 0.1078 FS FDR s.e. 0.0055 0.0106 0.0194 0.0092 0.0146 0.0230 0.0082 0.0096 0.0162 FS FNR s.e. 0.0195 0.0164 0.0106 0.0106 0.0126 0.0078 0.0256 0.0164 0.0118 T able 4. Comparison of nite horizon, rejectiv e step-up pro cedure for xed-sample equiv- alen ts where the LLR statistics are replaced with historical maxim ums, the n ull and alter- nativ e streams are indep enden t of eac h other, and eac h stream is p ositiv ely correlated with all others of its same v eracit y , as in Case 5. Again, p erformance at T = 50 is disapp oin ting, and inferior to the equiv alen t sequen tial step-do wn metho d. 2.5. CONCLUSIONS AND DISCUSSION 73 P oisson P oisson Unscaled Binomial (3, 7) (5, 5) (7, 3) (3, 7) (5, 5) (7, 3) (3, 7) (5, 5) (7, 3) H 0 ASN 49.5720 49.6828 49.7567 47.9003 49.2866 48.7687 49.8767 49.9308 49.9230 H 1 ASN 35.2806 36.5252 38.7810 27.9841 33.7082 32.8057 40.3840 41.0732 41.2883 Seq FDR 0.0105 0.0162 0.0264 0.0330 0.0284 0.0777 0.0051 0.0081 0.0101 Seq FNR 0.3357 0.2241 0.1355 0.2222 0.1946 0.0970 0.4889 0.3208 0.1790 Seq FDR s.e. 0.0014 0.0023 0.0035 0.0026 0.0030 0.0056 0.0014 0.0019 0.0020 Seq FNR s.e. 0.0077 0.0060 0.0042 0.0076 0.0060 0.0039 0.0055 0.0046 0.0034 FS FDR 0.0370 0.0471 0.0791 0.0729 0.1030 0.1785 0.0521 0.0433 0.0647 FS FNR 0.1476 0.0856 0.0839 0.0692 0.0420 0.0405 0.2008 0.1843 0.1062 FS FDR s.e. 0.0078 0.0104 0.0165 0.0105 0.0154 0.0223 0.0087 0.0104 0.0142 FS FNR s.e. 0.0222 0.0137 0.0109 0.0171 0.0098 0.0085 0.0230 0.0149 0.0115 T able 5. Comparison of nite horizon, rejectiv e step-up pro cedure for xed-sample equiv- alen ts where the LLR statistics are replaced with historical maxim ums, with T = 75 as in Case 6. Sa vings are signican t and error con trol is main tained handily . Finally a comparison of step-do wn with BL st yle cutos to step-up with BH-st yle cutos, otherwise similar to Cases 6 and 4. In T able 6 w e see that for the larger horizon T = 75, step-up tends to outp erform step-do wn, b oth with and without HM. (m 0;m 1 ) = (5; 5) P oisson P oisson Unscaled Binomial T = 75 SD BL SD BL HM SU BH SU BH HM SD BL SD BL HM SU BH SU BH HM SD BL SD BL HM SU BH SU BH HM H 0 ASN 74.9260 74.8546 73.7034 74.9564 73.7868 70.6702 71.9598 69.1206 74.9500 74.9446 74.3006 74.9088 H 1 ASN 56.2144 55.6298 38.4446 58.2552 42.7954 40.5948 33.6100 30.2606 61.3780 61.1274 43.4294 59.4144 Seq FDR 0.0031 0.0079 0.0304 0.0027 0.0348 0.1004 0.0609 0.1035 0.0024 0.0030 0.0189 0.0062 Seq FNR 0.2288 0.2149 0.1076 0.2723 0.0899 0.0719 0.0621 0.0503 0.3088 0.3029 0.1562 0.2674 Seq FDR s.e. 0.0009 0.0013 0.0024 0.0009 0.0025 0.0049 0.0030 0.0037 0.0012 0.0012 0.0019 0.0012 Seq FNR s.e. 0.0039 0.0042 0.0037 0.0032 0.0036 0.0035 0.0031 0.0030 0.0037 0.0039 0.0039 0.0043 FS FDR 0.0237 0.0258 0.0597 0.0457 0.1163 0.1076 0.1204 0.1214 0.0274 0.0363 0.0560 0.0616 FS FNR 0.0632 0.0618 0.0313 0.0451 0.0172 0.0154 0.0107 0.0257 0.1134 0.0987 0.0728 0.0696 FS FDR s.e. 0.0066 0.0072 0.0095 0.0084 0.0155 0.0137 0.0122 0.0112 0.0069 0.0085 0.0090 0.0108 FS FNR s.e. 0.0095 0.0099 0.0067 0.0088 0.0056 0.0056 0.0042 0.0071 0.0129 0.0123 0.0106 0.0107 T able 6. Comparison of nite horizon, rejectiv e step-do wn and step-up pro cedures against eac h other as w ell as their xed-sample equiv alen ts; b oth with and without HM, with T = 75 as in Case 6. Sa vings are signican t and error con trol is main tained handily . 2.5. Conclusions and Discussion Here w e presen ted a theorem for con trol of FDR without an y scaling under nite horizon rejectiv e pro cedures, sub ject to a n um b er of conditions. All but Condition 2.2 are trivial to either conrm or enforce, but that condition is dicult to v erify . Ho w ev er, w e ha v e sho wn that it is applicable to the nite horizon rejectiv e step-do wn Algorithm 1 in tro duced in Chapter 1 and the equiv alen t step-up algorithm in tro duced here when the historical maxim ums are used instead of the LLR statistics. Ho w ev er there ma y b e a n um b er of other conditions on the distributions and rejection pro cedures that w ould ensure satisfaction of Condition 2.2. After presen ting the theorem and the sequen tial step-up pro cedure, w e explored its p erformance on a n um b er of data generation sc hemes, including the Y ello w card data. W e found that the more our data 2.5. CONCLUSIONS AND DISCUSSION 74 streams conformed to Finner et al.’s conditions, the b etter our pro cedure p erformed. W e also compared it to equiv alen t settings for the sequen tial step-do wn pro cedure, whic h w e generally found to b e more ecien t. Our application of the four sequen tial testing pro cedures presen ted in this thesis to the sim ulated Y el- lo w card drug side eect rep orts indicates that it could b e emplo y ed b y the MHRA to automatically detect previously unrecognized adv erse drug reactions. The MultSeq soft w are pac k age discussed in Chapter 3, with its handling of streaming data, could b e installed on the MHRA serv er and set to regularly issue rep orts on new detections (as w ell as drugs v eried to b e unasso ciated with certain side eects) with minimal eort, to p oten tially great b enet. CHAPTER 3 MultSeq Python Pack age 3.1. In tro duction MultSeq is an extensible Python pac k age (Rossum, 1995, v ersion 3.5.3, 64-bit, on Ubun tu Lin ux) w e ha v e dev elop ed that implemen ts the sequen tial step-up and sequen tial step-do wn m ultiple h yp othesis testing pro cedures, as describ ed in Chapters 1 and 2. The main engine of the pac k age is in the MultSeq mo dule, whic h con tains a general sequen tial testing function that allo ws sp ecication of arbitrary rejection, acceptance, and termination criteria as w ell as the xed-sample step-up and step-do wn pro cedures on whic h the sequen tial pro cedure extensions rely; b oth nite horizon rejectiv e pro cedures and innite horizon acceptiv e-rejectiv e pro cedures are implemen ted and it can handle streaming data in addition to pregenerated data. The pac k age also con tains a range of functions for calculating cuto lev els, b oth in terms of p-v alues that satisfy FDR and FNR con trols under a v ariet y of testing pro cedures and dep endency assumptions, and in terms of lik eliho o d ratio cuto v alues corresp onding to a set of suc h p-v alue cutos. These lik eliho o d ratio cutos can b e calculated using a generalization of the W ald appro ximations or through Mon te Carlo (MC) sim ulation, b oth of whic h are discussed in Section A.1.3. The pac k age includes a data set of drug reaction rep orts mined from the UK’s Y ello w card Pharmaco vig- ilence Database (MHRA, 2017) as describ ed in Sections 1.4.2 and 2.4, and a n um b er of functions that allo w that data to b e used as the basis for sim ulating data streams to whic h the man y sequen tial testing pro cedures can b e applied, describ ed in Section 3.1.3.1. It also con tains Go ogle searc h data that acts as a secondary source against whic h the sim ulation results can b e compared via a wide range of built-in visualization options. This data w as collected via using Go ogle’s Python API (Go ogle, 2016) on April 10, 2016. W e c hose to use Python for a v ariet y of reasons, but primarily b ecause of its ease of dev elopmen t and the ric h library of useful computational pac k ages a v ailable. As a programming language dev elop ed b y a mathematician (v an Rossum, 2009), it is w ell suited to scien tic tasks, and has courted the op en source comm unit y eectiv ely , encouraging the dev elopmen t of pac k ages to matc h the to ols pro vided b y man y sp ecialized languages all in one language with excellen t in terop erabilit y . 3.1.1. Soft w are Dep endencies. MultSeq relies on a n um b er of pac k ages that ha v e b ecome standard parts of the Python scien tic library . The most fundamen tal of these are NumPy (for basic v ectorized math as 75 3.1. INTR ODUCTION 76 in MatLab ), SciPy (whic h includes a la y er of statistical and mathematical features la y ered on top of NumPy , mimic king man y of the higher lev el function in MatLab ) (V an Der W alt et al., 2016, NumPy v ersion 1.3.1, SciPy v ersion 0.19.1), and P andas (whic h pro vides ecien tly manipulated DataF rame ob jects, designed to em ulate Data.T able ’s in the R programming language) as describ ed in McKinney (2012, v ersion 0.20.3). F or visualizations, MultSeq emplo ys a com bination of the standard matplotlib plotting library (Hun ter, 2007, v ersion 2.0.2), reminiscen t of MatLab ’s standard plotting library , with the Seab o rn la y er W ask om, Mic hael and seab orn Dev elopmen t T eam (2014, v ersion 0.7.1) for impro v ed visualizations, and the Bok eh library whic h allo w for more complex, in teractiv e plots whic h can b e displa y ed in-bro wser. 3.1.2. Main Engine. The main engine is a function that p erforms a single instance of a sp ecied acceptance and rejection pro cedure (b y whic h w e mean it outputs a set of rejected h yp otheses and a set of accepted h yp otheses) on input data. It tak es as argumen ts either a P andas DataF rame ob ject con taining the v alues of log lik eliho o d ratio (LLR) statistics, with columns corresp onding to h yp otheses and ro ws corre- sp onding to time steps, or an online_data ob ject that yields streaming data, dened in the data mo dule and discussed in Section 3.1.3.1; a v ector of rejection cutos, of length equal to the total n um b er of h yp otheses; and an optional v ector of acceptance cutos, of the same length as the rejection cuto v ector. It also accepts a parameter sp ecifying whether step up or step do wn tests should b e p erformed. A t eac h time step, the engine retriev es the latest data from all activ e data streams; applies the acceptance test function to an y activ e h yp otheses and cuto v alues, terminating the data streams for an y h yp otheses that are accepted, remo ving the corresp onding cuto v alue from further consideration, and recording the step and lev el at whic h they w ere accepted; then do es the same for the rejection test. F or all of the sequen tial pro cedures discussed in this thesis the order of application of acceptance v ersus rejection decisions do es not matter b ecause of the fundamen tal assumption that the acceptance and rejection critical v alues do not o v erlap; see Sections 1.3.1 and 2.2. In the nite horizon rejectiv e settings, these steps are p erformed only for the rejections. Once either the nite horizon has b een reac hed or all h yp otheses ha v e terminated, the engine returns the follo wing op erating c haracteristics (1) Hyp othesis sp ecic outcomes (a) The sets of h yp otheses that w ere accepted, rejected, and failed to terminate. (b) A DataF rame with a ro w for eac h h yp othesis and columns sp ecifying termination step, termi- nation t yp e (accept, reject, non-termination), and termination lev el. (c) A DataF rame with a ro w for eac h cuto lev el and rejection and acceptance columns whose v alues are the time step at whic h that cuto w as triggered. 3.1. INTR ODUCTION 77 (2) Coarse grained diagnostic data (a) Time series whose gran ularit y ma y b e sp ecied b y the user indicating ho w man y h yp otheses had b een rejected and accepted at a regular time step in terv al. 3.1.3. Supp ort Utilities. In addition to the main engine, MultSeq comes with a v ariet y of utilit y mo dules used to set up the pro cedures for actual use, as w ell as to analyze and visualize their long-run op erating c haracteristics. These include mo dules for (1) Data mo dule: for reading and sim ulating data, (2) Cuto mo dule: for calculating error con trolling cuto v alues, (3) Simulation and Sim Analysis mo dules: for running full sim ulations and analyzing the results, (4) Visualization mo dule: for visualizing the op erating c haracteristics and path wise p erformance of the pro cedures. 3.1.3.1. Data Mo dule. The Data mo dule is mainly comprised of 4 distinct groups of functions, and a group of classes. The rst group of functions deal with constructing the h yp otheses and either reading in or generating the ground truth data. In the case of Binomial, P oisson, or Gaussian examples, it reads in the n um b er of n ull and alternativ e h yp otheses and the corresp onding Binomial prop ortions, P oisson rates, or Gaussian means. It returns a P andas Series with the named h yp otheses and corresp onding parameter v alues. F or sim ulations using the Y ello w card drug side eects data, it either reads in and returns the actual amnesia rates and non-amnesia reaction rates for eac h drug, or generates drug names with o v erall reaction rate on a linear gradien t and amnesia prop ortions corresp onding to the n ull or alternativ e h yp otheses, and returns b oth Series . The Gaussian h yp othesis generator tak es a gradien t of v ariances, assuming the least is 1.0, and the n ull and alternativ e means. There is also a function for generating P oisson parameters on a gradien t, without a corresp onding h yp othesis asso ciated to it. The second group of functions generates the sim ulated data for the data streams. They tak e in the parameter v ectors, the n um b er of time steps of data to generate, and an optional correlation co ecien t parameter, using them to generate the relev an t t yp es of samples (Binomial success coun ts, P oisson ev en t coun ts, Gaussian sample means and sample v ariance, or drug side eect coun ts), and returning the sucien t cum ulativ e statistic or statistics for eac h stream. The third group are functions that tak e the stream statistics emitted b y the group of functions de- scrib ed in the previous paragraph, along the n ull and alternativ e h yp othesis parameters, and return a P andas DataF rame of LLRs for eac h h yp othesis at eac h time step, as describ ed in Section 1.2.0.2. F or eac h supp orted data t yp e, the data mo dule also con tains an imp ortance sample w eigh t function that is used to calculate 3.1. INTR ODUCTION 78 corresp onding w eigh ts, as in Equation A.1.4 in Section A.1.3, for eac h LLR v alue for use in nite horizon cuto estimation, as describ ed in Section A.1.3. The fourth group are functions for screening drug data and dev eloping n ull and alternativ e h yp otheses to test based on the ra w reaction data, describ ed in Section 1.4.2. The nal group is a set of classes used to facilitate testing of truly streaming data. The primary class, online_data tak es as argumen ts an initial list of h yp otheses, and data generating function that tak es a list of activ e h yp otheses and returns the most up-to-date LLR statistics for eac h of them. The other classes are used to wrap a P andas DataF rame ob ject, so that it conforms to the online_data in terface. F or instance, a user migh t write some function that tak es the sym b ols for a set of publicly traded sto c ks, queries an online service for their recen t p erformance and a lo cal database for some parameter information ab out h yp otheses concerning eac h of them, then use it to return their LLRs. Initialized with a relev an t set of sto c ks to w atc h and left to run on a serv er, yielding a new time step ev ery da y , the user migh t quic kly whittle do wn whic h sto c ks to add to his or her p ortfolio. 3.1.3.2. Cuto Mo dule. The functions in this mo dule are used to pro vide the user with error con trolling cutos. They are organized in to three groups. The rst deals primarily with calculating the FDR b ound giv en a v ector of p-v alue rejection cutos, based on the w ork of Guo and Rao (2008), and in Section 1.3. This group includes functions that tak e suc h p-v alue v ectors and a desired FDR b ound and return adequately scaled v ectors of p-v alue cutos. There are also help er functions that tak e the n um b er of h yp otheses and desired FDR con trol lev el and return the corresp onding Benjamini-Ho c h b erg (as in Equation 1.2.6 in Chapter 1) and Benjamini-Liu (as in Equation 1.2.7 in Chapter 1) st yle p-v alue cuto v alues, b oth scaled (using the Guo-Rao factor from Equation 1.3.5 in Chapter 1) and unscaled (for use with indep enden t data streams). There is also a more complex, iterativ e approac h to dev eloping pFDR and pFNR con trolling p-v alue v ectors. This is required b ecause the adequately scaled v ector relies up on the adequately scaled v ector, due to the probabilit y of an y acceptances and rejections resp ectiv ely , as sho wn in the discussion follo wing Theorem 1.4 . It should b e noted that an y function description referring only to FDR can b e generalized to FNR and acceptance cutos directly , as long as the and v ectors are suc h that the rejection regions ha v e no o v erlap with the acceptance regions. The next group deals with the translation of cuto v ectors b et w een p-v alue space and LLR space. F or innite horizon settings, this is accomplished via extensions of the W ald appro ximation dev elop ed b y Bartro (2014), and in v ersions thereof. F or nite horizon rejectiv e settings, the cutos are calculated via MC sim ulation, as this allo ws for use of more general statistics and is less conserv ativ e than the A i = log i v alues obtained b y taking the limit of the generalized W ald appro ximations as ! 1. This is done b y 3.1. INTR ODUCTION 79 (1) assuming that all the n ull h yp otheses are true, (2) for eac h MC rep etition sim ulating uncorrelated data from eac h of them, (3) con v erting them in to LLRs, (4) recording the maxim um LLR v alue for eac h h yp othesis and eac h MC rep etition o v er all time steps, (5) taking the 1~ quan tiles o v er those maxim um v alues for eac h stream, (6) and nally taking the maxim um o v er all streams for eac h cuto lev el. This is computationally exp ensiv e but due to its em barrassingly parallel nature, it can b e sp ed up signican tly b y passing the parallel computation ag, whic h spreads steps 1 through 4 o v er all the a v ailable pro cessors, and b y using imp ortance sampling, describ ed in Section A.1.3. A dditionally , there is an exp erimen tal MC-based approac h to rening innite horizon rejectiv e and acceptiv e LLR cuto v alues. It is implemen ted b y taking the maxim um v alue of the statistics o v er all time steps in the stream o ccurring b efore the statistic hits the most extreme acceptance v alue, assumed to b e negativ e. The same approac h is tak en to up date the acceptance cutos, wherein the minim um v alue reac hed b efore hitting the most extreme rejection v alue is recorded in place of the maxim um. Because rejectiv e cutos dep end on the acceptiv e cutos and vice v ersa, as in the marginal conditions in Equations 1.3.7 and 1.3.8, this m ust b e done iterativ ely . The W ald appro ximations are used as initial v alues, allo wing for quic k con v ergence to sligh tly less conserv ativ e v alues than the initial W ald appro ximations, at the cost of signican t initial computational cost. F or an y of the MC-based cuto estimation routines to ac hiev e decen t gran ularit y , w e suggest a bare minim um of the in v erse of the smallest dierence b et w een adjacen t elemen ts of the v ector as the n um b er of MC rep etitions when not using imp ortance sampling, for the reasons describ ed in Section A.1.3. When imp ortance sampling is emplo y ed, w e suggest using a n um b er of rep etitions an order of magnitude less than that in v erse, although smaller n um b ers ha v e suced in our exp erience. The nal group of functions estimates the termination time for a pro cedure an innite horizon, acceptiv e- rejectiv e pro cedure. T o do so, w e rst need to calculate the exp ected v alue of the marginal (or incremen tal) log lik eliho o d of one time step. The rst group of functions the mean and v ariance of eac h incremen t of the LLR, assumed to b e the sum of indep enden t, iden tically distributed incremen ts; the v ariance ma y b e useful when the Bro wnian Motion appro ximation is desired. The most imp ortan t function in this group then lev erages the exp ected LLR in com bination with the LLR cuto v ectors to estimate pro cedure termination time using a v arian t of the a v erage stopping time form ula for single h yp othesis sequen tial testing presen ted in of Siegm und (Equations 2.21 and 2.22 1985). This is also used to help determine ho w man y time steps to generate for the innite horizon sim ulations. 3.1. INTR ODUCTION 80 Siegm und (1985, Remark 2.20, page 13) presen ts the follo wing exp ected stopping time under the n ull and alternativ e h yp otheses, where 0 and 1 are the exp ected single time step LLR v alues under then n ull and alternativ e h yp otheses: E 0 [] = log 1 + (1) log 1 = 0 E 1 [] = (1) log 1 + log 1 = 1 : If w e assume that + < 1 thenE 1 [] is decreasing in b oth and , whereasE 0 [] in increasing in b oth. W e therefore adapt these estimates to the m ultiple h yp othesis setting using the follo wing conserv ativ e b ounds, and c ho ose the larger one as our naiv e exp ected termination time: (3.1.1) E 0 [] m log 1 m m + (1 m ) log m 1 m = 0 (3.1.2) E 1 [] (1 1 ) log 1 1 1 + 1 log 1 1 1 = 1 : So T 1 := maxfE 0 [];E 1 []g. In the m ultiple testing setting of the pro cedures describ ed in Chapters 1 and 2, w e redene = max i2f1;:::;mg i to b e the pro cedure termination time and assume that its exp ectation is less than T 1 . W e wish to limit under-generation of data probabilistically to some v alue c2 (0; 1), and set T gen = T 1 =c, then b y Mark o v’s inequalit y w e ha v e P (T gen )E []=T gen T 1 = (T 1 =c) =c: 3.1.3.3. Simulation Mo dule. The Sim ulation Mo dule pro vides t w o main out w ard facing functions for running the MultSeq pro cedure on sim ulated data, one generates the data from Y ello w card Drug Reactions data, and the other generates en tirely syn thetic Binomial, P oisson, Gaussian, or Drug Reaction-lik e data. Both of these functions run a user sp ecied n um b er of MC rep etitions of the sim ulation in parallel across 3.1. INTR ODUCTION 81 as man y cores as the mac hine has a v ailable, and handle b oth innite horizon and nite horizon rejectiv e scenarios. Both functions include a wide arra y of options, some of the most imp ortan t b eing: FDR and FNR con trol lev els, data correlation v alues and correlation structures, n ull and alternativ e h yp othesis sp ecications (or options for automatic h yp othesis sp ecication), nite horizon, pseudo-innite horizon undersho ot probabilit y (as describ ed in the previous section), and MC calculated cuto sub-gran ularit y probabilit y . The last t w o of whic h use conserv ativ e Mark o v inequalities. The Sim ulation Analysis Mo dule analyzes the results of the sim ulations describ ed ab o v e b y comparing them against xed-sample equiv alen ts, as in Section 1.4.1.3 and Section 2.4.2. 3.1.3.4. Visualization Mo dule. The visualization mo dule con tains a n um b er of plotting metho ds for visu- alizing the results of sim ulation outcomes. The most useful one generates an in teractiv e scatter plot using b ok eh , with one or more datasets, and allo ws for coloring and to oltips for individual datap oin ts. These plots are output to a HTML do cumen t, whic h can include data tables as w ell. Figure 3.1.1 sho ws suc h a plot, with a to oltip displa y ed o v er the glyph represen ting a the op erating c haracteristic of the t w o pro cedures on a particular h yp othesis. Figure 3.1.1. b ok eh scatter plot with to oltip. 3.2. DISCUSSION 82 The other plotting function plots the LLR paths of all of the h yp otheses, as w ell as all of the rejection (and acceptance) cutos, indicating when a h yp othesis has b een terminated or a cuto tripp ed b y plotting it as a dotted line from there on out. If ground truth is kno wn, then annotations regarding correct and incorrect determinations ma y b e applied, as in Figure 3.1.2. Figure 3.1.2. LLR paths and cutos. 3.2. Discussion MultSeq pro vides a ric h set of to ols for scien tists to apply to real problems, b oth in online and oine settings. With the recen t addition of supp ort for streaming data, as discussed at the end of Section 3.1.3.1, and its easily extensible design, MultSeq has the p oten tial to dev elop a user base of b oth statisticians and general researc hers. Prior to its full release, a rough demonstration v ersion will b e a v ailable for do wnload at http://www- scf.usc.edu/~mhankin/ b y mid August, 2017. Bibliography Bartro, J. (2014). Multiple h yp othesis tests con trolling generalized error rates for sequen tial data. arXiv pr eprint arXiv:1406.5933 . Bartro, J. and Lai, T. L. (2008). Ecien t adaptiv e designs with mid-course sample size adjustmen t in clinical trials. Statistics in me dicine , 27(10):15931611. Bartro, J. and Lai, T. L. (2010). Multistage tests of m ultiple h yp otheses. Communic ations in Statistics- The ory and Metho ds , 39:15971607. Bartro, J., Lai, T. L., and Shih, M.-C. (2012). Se quential exp erimentation in clinic al trials: design and analysis , v olume 298. Springer Science & Business Media. Bartro, J. and Song, J. (2013). Sequen tial tests of m ultiple h yp otheses con trolling false disco v ery and nondisco v ery rates. arXiv pr eprint arXiv:1311.3350 . Bartro, J. and Song, J. (2014). Sequen tial tests of m ultiple h yp otheses con trolling t yp e i and ii familywise error rates. Journal of statistic al planning and infer enc e , 153:100114. Bartro, J. and Song, J. (2015). A rejection principle for sequen tial tests of m ultiple h yp otheses con trolling familywise error rates. Sc andinavian Journal of Statistics . Benjamini, Y. and Ho c h b erg, Y. (1995). Con trolling the false disco v ery rate: a practical and p o w erful approac h to m ultiple testing. Journal of the R oyal Statistic al So ciety. Series B (Metho dolo gic al) , pages 289300. Benjamini, Y. and Liu, W. (1999). A step-do wn m ultiple h yp otheses testing pro cedure that con trols the false disco v ery rate under indep endence. Journal of Statistic al Planning and Infer enc e , 82(1):163170. Benjamini, Y. and Y ekutieli, D. (2001). The con trol of the false disco v ery rate in m ultiple testing under dep endency . A nnals of statistics , pages 11651188. Bennett, C. M., W olford, G. L., and Miller, M. B. (2009). The principled con trol of false p ositiv es in neuroimaging. So cial c o gnitive and ae ctive neur oscienc e , 4(4):417422. Bonferroni, C. E. (1936). T e oria statistic a del le classi e c alc olo del le pr ob abilita . Libreria in ternazionale Seeb er. Casella, G. and Berger, R. L. (2002). Statistic al infer enc e , v olume 2. Duxbury P acic Gro v e, CA. 83 Bibliograph y 84 De, S. K. and Baron, M. (2012). Sequen tial b onferroni metho ds for m ultiple h yp othesis testing with strong con trol of family-wise error rates i and ii. Se quential A nalysis , 31(2):238262. De, S. K. and Baron, M. (2015). Sequen tial tests con trolling generalized familywise error rates. Statistic al Metho dolo gy , 23:88102. Efron, B. and Tibshirani, R. J. (1994). A n intr o duction to the b o otstr ap . CR C press. eHealthMe, F. (2017). ehealthme drug side eect rep orts. data retriev ed from http://www.ehealthme.com/ ds/indomethacin/amnesia . Finner, H., Dic khaus, T., and Roters, M. (2009). On the false disco v ery rate and an asymptotically optimal rejection curv e. The A nnals of Statistics , 37(2):596618. Glynn, P . W. (1996). Imp ortance sampling for mon te carlo estimation of quan tiles. In Mathematic al Metho ds in Sto chastic Simulation and Exp erimental Design: Pr o c e e dings of the 2nd St. Petersbur g W orkshop on Simulation , pages 180185. Publishing House of St. P etersburg Univ ersit y . Go ogle (2016). Go ogle p ython api. Go vindara julu, Z. (1975). Se quential statistic al pr o c e dur es . A cademic Press. Guo, W. and Rao, M. B. (2008). On con trol of the false disco v ery rate under no assumption of dep endency . Journal of Statistic al Planning and Infer enc e , 138(10):31763188. Gur, R. H. H. (2012). F alse disco v ery rate con trolling pro cedures for discrete tests. arXiv pr eprint arXiv:1112.4627 . Ho c h b erg, Y. (1988). A sharp er b onferroni pro cedure for m ultiple tests of signicance. Biometrika , 75(4):800 802. Holm, S. (1979). A simple sequen tially rejectiv e m ultiple test pro cedure. Sc andinavian journal of statistics , pages 6570. Hun ter, J. D. (2007). Matplotlib: A 2d graphics en vironmen t. Computing In Scienc e & Engine ering , 9(3):9095. Ja v anmard, A. and Mon tanari, A. (2017). Online rules for con trol of false disco v ery rate and false disco v ery exceedance (to do: Needs up date). A nnals Of Statistics . McKinney , W. (2012). Python for data analysis: Data wr angling with Pandas, NumPy, and IPython . " O’Reilly Media, Inc.". MHRA, U. (2017). Y ello w card pharmaco vigilence database. data retriev ed from Y ello w card Pharmaco vigi- lence Database, https://yellowcard.mhra.gov.uk/ . Rossum, G. (1995). Python reference man ual. T ec hnical rep ort, Amsterdam, The Netherlands, The Nether- lands. Siegm und, D. (1985). Se quential analysis: tests and c ondenc e intervals . Springer Science & Business Media. Bibliograph y 85 Song, Y. and F ellouris, G. (2016). Sequen tial m ultiple testing with generalized error con trol: an asymptotic optimalit y theory . arXiv pr eprint arXiv:1608.07014 . Song, Y., F ellouris, G., et al. (2017). Asymptotically optimal, sequen tial, m ultiple testing pro cedures with prior information on the n um b er of signals. Ele ctr onic Journal of Statistics , 11(1):338363. Storey , J. D. (2002). A direct approac h to false disco v ery rates. Journal of the R oyal Statistic al So ciety: Series B (Statistic al Metho dolo gy) , 64(3):479498. Storey , J. D. (2003). The p ositiv e false disco v ery rate: a ba y esian in terpretation and the q-v alue. A nnals of statistics , pages 20132035. T riv edi, P . K., Zimmer, D. M., et al. (2007). Copula mo deling: an in tro duction for practitioners. F oundations and T r ends R in Ec onometrics , 1(1):1111. V an Der W alt, S., Colb ert, S., and V aro quaux, G. (2016). The n ump y arra y: a structure for ecien t n umerical computation (2011). v an Rossum, G. (2009). Python history . W ald, A. (1973). Se quential analysis . Courier Corp oration. W ask om, Mic hael and seab orn Dev elopmen t T eam (2014). se ab orn: statistic al data visualization . Wijsman, R. A. (1985). A useful inequalit y on ratios of in tegrals, with application to maxim um lik eliho o d estimation. Journal of the A meric an Statistic al Asso ciation , 80(390):472475. Wikip edia (2017). Y ello w card sc heme wikip edia, the free encyclop edia. [Online; accessed 11-August-2017 ]. APPENDIX A Appendix for Chapter 1 A.1. Cuto Estimation and Notes A.1.1. W ald Appro ximations for Single Hyp othesis SPR T T ests. In this section w e dev elop W ald’s appro ximations for sequen tial testing of simple h yp otheses; see also (Go vindara julu, 1975, p.20-21). In the sequen tial single simple h yp othesis testing framew ork w e tak e B <A and (t) = t X i=1 log f 1 (x i ) f 0 (x i ) Then the rejection rule is f9t s.t. (t)A\8t 0 <t (t 0 )>Bg: A cceptance is f9t s.t. (t)B\8t 0 <t (t 0 )<Ag: F or giv en ; > 0 w e wish to nd A and B suc h that P H0 (rejectH 0 ) =P H0 (9t s.t. (t)A\8t 0 <t (t 0 )>B) = P H1 (acceptH 0 ) =P H0 (9t s.t. (t)B\8t 0 <t (t 0 )<A) = No w dene E t R t suc h that if (X 1 ;:::X t )2 E t then H 0 is rejected at the t th time step, and F t R t b e the set suc h that (X 1 ;:::X t )2 F t then H 0 is accepted at the t th time step. Clearly8s;t2 N s.t. s < t (E s R ts )\E t =;, i.e.fE t g t2N andfF t g t2N are all m utually disjoin t. This giv es P H0 (rejectH 0 ) = X t P H0 (E t ) =P H0 ([ t E t ) = P H1 (rejectH 0 ) = X t P H1 (E t ) =P H1 ([ t E t ) = 1 If w e assume that the pro cess terminates, then w e also ha v e 86 A.1. CUTOFF ESTIMA TION AND NOTES 87 P H0 (([ t E t )[ ([ t F t )) =P H1 (([ t E t )[ ([ t F t )) = 1 Clearly in the rejection region w e m ust ha v e e A f 1 ( ~ X)f 0 ( ~ X) so, with sligh t abuse of notation: = 1 X t=1 P H0 (E t ) = 1 X t=1 Z Et f 0 ( ~ X)d ~ X 1 X t=1 e A Z Et f 1 ( ~ X)d ~ X =e A 1 X t=1 P H1 (E t ) =e A P H1 (rejectH 0 ) =e A (1) Similarly , w e ha v e f 1 ( ~ X)e B f 0 ( ~ X), and th us = 1 X t=1 P H1 (F t ) = 1 X t=1 Z Ft f 1 ( ~ X)d ~ X 1 X t=1 e B Z Ft f 0 ( ~ X)d ~ X =e B 1 X t=1 P H0 (F t ) =e B P H0 (acceptH 0 ) =e B (1) In order to nd the optimal A and B w e searc h the b oundary b y setting the inequalities to equalities, and obtain A = log 1 + B = log 1 Bartro (2014) sho ws that this can b e generalized to the m ultiple h yp othesis setting A j = log j (1 1 ) 1 1 1 (1 j ) + A.1. CUTOFF ESTIMA TION AND NOTES 88 B j = log 1 1 1 (1 j ) j (1 1 ) Where is a correction term that helps to correct for the appro ximateness of the W ald appro ximations, and is usually set to = 0:583. A.1.2. Bartro Multiple Hyp othesis W ald Appro ximations. T ak en directly from Theorem 5.1 of Bartro (2014). Dene A (a;b) = log 1b a and B (a;b) = log b 1a + ~ j = 1 (1 j ) 1 1 and ~ j = 1 (1 j ) 1 1 ; and use those denitions to further dene A j :=A j ; ~ j andB j :=B (~ j ; j ): No w for all j2f1;:::;mg the v alues A (~ j ; j ) = log (1 j ) (1 j ) (1 1 ) 1 = log 1 1 1 = log (1 1 ) (1 1 ) (1 1 ) 1 =A 1 ; ~ 1 =A 1 are equiv alen t, as are the v alues of A.1. CUTOFF ESTIMA TION AND NOTES 89 B j ; ~ j = log (1 j ) (1 j ) (1 1 ) 1 + = log 1 1 1 + = log (1 1 ) (1 1 ) (1 1 ) 1 + =B (~ 1 ; 1 ) =B 1 : No w examining the marginal constrain t conditions and applying these denitions, equiv alencies, and the W ald appro ximations, w e ha v e P 0 (9t<1 s.t. (t)A j \8t 0 <t (t 0 )>B 1 ) =P 0 9t<1 s.t. (t)A j ; ~ j \8t 0 <t (t 0 )>B j ; ~ j = j and P 1 (9t<1 s.t. (t)B j \8t 0 <t (t 0 )<A 1 ) =P 1 (9t<1 s.t. (t)B (~ j ; j )\8t 0 <t (t 0 )<A (~ j ; j )) = j : A.1.3. Finite Horizon Cuto Calculation via Mon te Carlo and Imp ortance Sampling Meth- o ds. Using the data a v ailable ab out eac h of the streams, sim ulate T p erio ds of data from eac h of them according to their n ull h yp otheses, and rep eat this k reps times. F or eac h rep etition and eac h stream, record the maxim um v alue of the test statistic. Then for eac h stream i and eac h j cuto lev el, nd the 1 th j - quan tile across all sim ulations, whic h w e will temp orarily refer to as A i j . No w c ho ose A j = max i A i j , then 8im P H i 0 (9t<T s.t. i (t)A j )P H i 0 (9t<T s.t. i (t)A i j ) j : F or the Y ello w card drug data, tak e X i t = (Y i t ;Z i t ) where Y i t is the n um b er of new amnesia side eect rep orts at time t and Z i t is the n um b er of new non-amnesia side eect rep orts at time t. T ak e i to b e the A.1. CUTOFF ESTIMA TION AND NOTES 90 total side eect rep ort P oisson rate for the i th drug, computed b y summing the amnesia-related side eects rate with the non-amnesia side eects rate. F or eac h MC sim ulation, w e generate X i t with Y i t Pois( i p 0 ) and Z i t Pois( i (1p 0 )) and use those to compute the LLR statistic (A.1.1) i (t) = t X s=1 Y i s log(p 1 =p 0 ) +Z i s log((1p 1 )=(1p 0 )) for t T . F or eac h MC sim ulation b tak e i b = max tT i (t), o v er the set of statistic v alues for that h yp othesis in that sim ulation. No w set A i j to the 1 j quan tile of i b bBreps , and A j = max im A i j . The smaller the cutos, the more MC iterations are required to accurately estimate the quan tiles. T o see this, note that if 1 = 0:1 and 2 = 0:2, w e m ust generate at least 10 = 1= ( 2 1 ) samples to distinguish A 1 from A 2 without in terp olation. F or particularly conserv ativ e t yp e 1 error con trols, standard MC approac hes ma y end up b eing computationally infeasible. W e therefore emplo y imp ortance sampling metho ds for quan tile estimation, as describ ed in Glynn (1996), in order to reduce the v ariance of our estimates with far few er samples. F or a simple n ull h yp othesis, tak e 2H i 0 . W e wish to estimate (A i ) i2f1;:::;mg suc h that 1 i =F (A i ) for i2f1;:::;mg where F (a) =P max tT (t)a : No w, for an y 0 2 i w e ha v e F (a) =P max tT (t)a =E 1 max tT (t)a =E 0 1 max tT (t)a dP dP 0 : W e ma y estimate this v alue via MC sim ulation with lo w er v ariance as follo ws: E 0 1 max tT (t)a dP dP 0 1 B B X b=1 1 max tT X b 1 ;:::;X b t a dP dP 0 X b 1 ;:::;X b T X b 1 ;:::;X b T P 0: In practice, to ensure a v alid CDF w e dene A.1. CUTOFF ESTIMA TION AND NOTES 91 ^ F (a) := B X b=1 1 max tT X b 1 ;:::;X b t a dP dP 0 X b 1 ;:::;X b T ! = B X b=1 dP dP 0 X b 1 ;:::;X b T ! : T o c ho ose our sampling distribution, w e shift the n ull distribution in the direction of the alternativ e, and often b ey ond. F or eac h MC sim ulation b B w e dra w X b 1 ;:::;X b t from the imp ortance sampling distribution, using a scaling factorS . In the Binomial case where w e are testing H 0 :p =p 0 vs.H 1 :p =p 1 , w e generate from (A.1.2) p = sigmoid (logit (p 0 ) +S (logit (p 1 ) logit (p 0 ))) ; in the P oisson case w e use = exp (log ( 0 ) +S (log ( 1 ) log ( 0 ))) ; for Gaussian H 0 :N 0 ; 2 vs.H 1 :N 1 ; 2 w e use N 0 +S ( 1 0 ); 2 ; nally , for the drug reactions where H 0 : Y P oisson (p 0 ); Z P oisson ( (1p 0 )) vs.H 0 : Y P oisson (p 1 ); Z P oisson ( (1p 1 )), w e hold constan t and use the same scaling approac h for p as in Equation A.1.2. Once the data has b een sim ulated, w e compute the maxim um LLR v alue and a w eigh t v alue: (A.1.3) b = max tT X b 1 ;:::;X b t (A.1.4) ~ L b = dP dP 0 X b 1 ;:::;X b T : After all the sim ulations ha v e b een conducted, w e reorder the v ector of pairs b ; ~ L b B b=1 in increasing order of the b ’s, and then normalize the w eigh ts L b = ~ L b = P B b 0 =1 ~ L b 0 to appro ximate the v alues P = b L b and th us P b P b b 0 =1 L b 0 . W e no w ha v e b ;L b B b=1 with L B = 1. In order to estimate the upp er th quan tile w e dev elop a linear in terp olation function ^ h (`) from the input L b B b=1 and the resp onse b B b=1 , and pass it the argumen t ` = 1. The reader will no doubt notice that this ignores some of the b oundary issues, but w e ha v e found that it is the most computationally reasonable solution to our problem. A.2. EXPLICIT STOPPING TIMES 92 A.1.4. pFDR Con trol Estimation. In Theorem 1.3, pFDR is sho wn to b e con trolled at D (m 0 ;m 1 ;~ )=P 0 (R> 0): Giv en that P 0 (R> 0) itself dep ends on ( i ) i2f1;:::;mg , MC tec hniques ma y b e required to determine the actual lev el of pFDR con trol. T o do so, w e rst lo w er b ound P 0 (R> 0) as P 0 (R> 0) =P 0 [ m i=1 reject H i 0 max i2f1;:::;mg P 0 reject H i 0 max i2f1;:::;mg P 0 9t<T s.t. i (t)A m max i2f1;:::;mg min 2 P 9t<T s.t. i (t)A m : Unfortunately , the nal quan tit y ma y b e dicult to estimate, as ma y b e innite. When m 0 = 0, pFDR con trol is mo ot. Ho w ev er, when m 0 1 w e also ha v e max i2f1;:::;mg P 0 9t<T s.t. i (t)A m min i2f1;:::;mg min 2H i 0 P 9t<T s.t. i (t)A m ; whic h is simple to estimate when H i 0 is nite or unitary (i.e. simple). A.2. Explicit Stopping Times Stopping times for Algorithms 1 and 2 can b e presen ted explicitly . F or the nite horizon, rejectiv e algorithm w e ha v e i = minft 1 : 9j2f1;:::;mg s.t. i (t)>A j \ jfi 0 2f1;:::;mgnfig : i 0tgjj 1g: F or innite horizon, acceptiv e-rejectiv e algorithm w e ha v e: A.3. BH CUTOFF EXAMPLE 93 i =ft 1 : 9j2f1;:::;mg s.t. i (t)>A j \ j n i 0 2f1;:::;mgnfig : i 0t\ i 0 ( i 0)> 0 o jj 1 [ i (t)<B j \ j n i 0 2f1;:::;mgnfig : i 0t\ i 0 ( i 0)< 0 o jj 1g: A.3. BH Cuto Example W e presen t here applications of Theorems 1.1 and 1.2 to BH st yle cutos, as describ ed in Equation 1.2.6, as an elegan t visualization of the form of the Guo-Rao scaling function. Cor ollar y A.1. FDR Contr ol for Finite Horizon R eje ctive Se quential Step-down Pr o c e dur e with BH-Style Cutos Given (1.3.6) wher e i = ~ i=m for ~ 2 (0; 1), we may b ound FDR of A lgorithm 1, under the true distribution, r e gar d less of its dep endenc e structur e, as fol lows: FDR ~ max 1m0m m1=mm0 m 0 m1+1 X k=1 1 k + m X k=m1+2 m 1 k (k 1) ! : The innite horizon equiv alen t theorem is presen ted b elo w. Cor ollar y A.2. FDR and FNR Contr ol for Innite Horizon with BH-Style Cutos Given (1.3.7) and (1.3.8) wher e i = ~ i=m and i = ~ i=m for ~ ; ~ 2 (0; 1), we may b ound FDR and FNR of A lgorithm 2, under the true distribution, r e gar d less of its dep endenc e structur e, as fol lows: FDR ~ max 1m0m m1=mm0 m 0 m1+1 X k=1 1 k + m X k=m1+2 m 1 k (k 1) ! FNR ~ max 1m0m m1=mm0 m 1 m0+1 X k=1 1 k + m X k=m0+2 m 0 k (k 1) ! : APPENDIX B Appendix for Chapter 2 B.1. PRDS Equalit y-Inequalit y Lemma T o sho w that PRDS on the statistics implies that the same conditionals on the P (Dj max tT i (t)x) is also non-decreasing in x, w e require the follo wing lemma. Lemma B.1. PRDS c onditione d on e quality implies PRDS c onditione d on gr e ater than If for al l sets D , i2 I 0 we have that P (DjX i =x) is non-de cr e asing in x, then P (DjX i x) is also non-de cr e asing in x. W e use a simplication of Wijsman’s pro of to pro v e this sp ecial case of his more general theorem.(Wijsman, 1985) Pr oof. Assume y>x, and tak e s x (u) = 1 ux to b e the hea viside function. No w dene F (u;v) :=s x (u)s y (v)s x (v)s y (u) G(u;v) :=f X (u)f XjD (v)f X (v)f XjD (u): Then w e ha v e F < 0 , ((u<x)[ (v<y))\ (vx)\ (uy) , (uy)\ (xv<y) )v<u; with the second equiv alence due to our assumption of y>x. A dditionally , 94 B.1. PRDS EQUALITY-INEQUALITY LEMMA 95 (v<u)\ (s x (u)s y (v) = 1) )x<yv<u ) (xv)\ (yu) )s x (v)s y (u) = 1 )F = 0; and clearly s x (u)s y (v) = 0)F 0; so w e ha v e v<u)F 0. The sign of G(u;v) is equiv alen t to the sign of G(u;v)P(D) f X (u)fx(v) G(u;v)P (D) f X (u)f x (v) = f XjD (v)P (D) f X (v) f XjD (u)P (D) f X (u) =P (DjX =v)P (DjX =u) so b y PRDS v<u)G(u;v) 0 and G(u;v)< 0)v<u. Com bining these facts w e can see that if v<u then F;G 0 soFG 0, and if vu F;G 0, so once again FG 0. Th us8u;v F (u;v)G(u;v) 0 and therefore 1 2 RR F (u;v)G(u;v)dudv 0 B.1. PRDS EQUALITY-INEQUALITY LEMMA 96 1 2 Z Z F (u;v)G(u;v)dudv = 1 2 Z Z (s x (u)s y (v)s x (v)s y (u)) f X (u)f XjD (v)f X (v)f XjD (u) dudv = 1 2 Z Z s x (u)s y (v)f X (u)f XjD (v)s x (v)s y (u)f X (u)f XjD (v) + s x (v)s y (u)f X (v)f XjD (u)s x (u)s y (v)f X (v)f XjD (u) dudv = Z Z s x (u)s y (v)f X (u)f XjD (v)s x (v)s y (u)f X (u)f XjD (v)dudv = Z s x (u)f X (u)du Z s y (v)f XjD (v)dv Z s y (u)f X (u)du Z s x (v)f XjD (v)dv = Z 1 x f X (u)du Z 1 y f XjD (v)dv Z 1 y f X (u)du Z 1 x f XjD (v)dv =P (Xx)P (XyjD)P (Xy)P (XxjD) = P (Xy)P (Xx) P (D) P (Xy;D) P (Xy) P (Xx;D) P (Xx) = P (Xy)P (Xx) P (D) (P (DjXy)P (DjXx)) P (DjXy)P (DjXx) = P (D) P (Xy)P (Xx) 1 2 Z Z F (u;v)G(u;v)dudv 0 Th us P (DjXy)P (DjXx) 0 and w e ha v e that P (DjXx) is non decreasing in x. APPENDIX C Appendix for Chapter 3 C.1. Roadmap Plans for further dev elopmen t of MultSeq; in order of decreasing priorit y (1) F ully do cumen ted GitHub release with unittests for all ma jor functions. (a) Standardized function naming. (2) F urther dev elopmen t of the data mo dule is in progress with the goal of directly in tegrating a sim ulation conguration language and standardized record storage pro cedure. A dditionally , there are plans for a greater degree of mo dularization whic h should allo w for more general data generation sc hemes as w ell as statistical testing sc hemes. (3) Other rejection routines (a) F ellouris and Song et al. (2017)’s sync hronous stopping routines. (b) Hankin’s adaptation of the aforemen tioned sync hronous stopping routines to -FDP and -FNP con trol. (4) Impro v ed m ultipro cessing and standardized c hec kp oin ting (5) More ecien t step-up 97
Abstract (if available)
Abstract
We explore the convergence of sequential testing and multiple hypothesis testing under FDR and FDR-like error controls, with arbitrary or relaxed conditions on the joint distribution. We present both theoretical and applied results. Additionally, we provide and discuss a software package for running the routines described in this publication.
Linked assets
University of Southern California Dissertations and Theses
Conceptually similar
PDF
Sequential testing of multiple hypotheses
PDF
Asymptotically optimal sequential multiple testing with (or without) prior information on the number of signals
PDF
Finite sample bounds in group sequential analysis via Stein's method
PDF
Evaluation of sequential hypothesis tests for cross validation of learning models using big data
PDF
Applications of Stein's method on statistics of random graphs
PDF
Large-scale inference in multiple Gaussian graphical models
PDF
Topics in selective inference and replicability analysis
PDF
Statistical inference for second order ordinary differential equation driven by additive Gaussian white noise
PDF
Model selection principles and false discovery rate control
PDF
Reproducible large-scale inference in high-dimensional nonlinear models
PDF
Methodology and application of modern genetic association tests in admixed populations
PDF
Interval arithmetic and an application in finance
PDF
Recurrent neural networks with tunable activation functions to solve Sylvester equation
PDF
Conditional mean-fields stochastic differential equation and their application
PDF
High-dimensional feature selection and its applications
PDF
Assessing cerebral oxygen supply and demand in sickle cell anemia using MRI
PDF
Dynamic network model for systemic risk
PDF
Controlled McKean-Vlasov equations and related topics
PDF
Facial key points detection by convolutional neural network
PDF
Big data analytics in metagenomics: integration, representation, management, and visualization
Asset Metadata
Creator
Hankin, Michael Ethan
(author)
Core Title
Sequential testing of multiple hypotheses with FDR control
School
College of Letters, Arts and Sciences
Degree
Doctor of Philosophy
Degree Program
Applied Mathematics
Publication Date
11/06/2017
Defense Date
09/18/2017
Publisher
University of Southern California
(original),
University of Southern California. Libraries
(digital)
Tag
false discovery rate,multiple testing,OAI-PMH Harvest,sequential probability ratio test,sequential testing
Language
English
Contributor
Electronically uploaded by the author
(provenance)
Advisor
Bartroff, Jay (
committee chair
), Lv, Jinchi (
committee member
), Rosen, Gary (
committee member
)
Creator Email
meh2135@gmail.com,mhankin@google.com
Permanent Link (DOI)
https://doi.org/10.25549/usctheses-c40-451028
Unique identifier
UC11264286
Identifier
etd-HankinMich-5880.pdf (filename),usctheses-c40-451028 (legacy record id)
Legacy Identifier
etd-HankinMich-5880.pdf
Dmrecord
451028
Document Type
Dissertation
Rights
Hankin, Michael Ethan
Type
texts
Source
University of Southern California
(contributing entity),
University of Southern California Dissertations and Theses
(collection)
Access Conditions
The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...
Repository Name
University of Southern California Digital Library
Repository Location
USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA
Tags
false discovery rate
multiple testing
sequential probability ratio test
sequential testing