University of Southern California Dissertations and Theses

Asymptotically optimal sequential multiple testing with (or without) prior information on the number of signals

(USC Thesis Other)

Asymptotically optimal sequential multiple testing with (or without) prior information on the number of signals

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Asymptotically optimal sequential multiple testing with (or without) prior information on the number of signals Author: Xinrui He Supervisor: Jay Bartroff A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA in Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (APPLIED MATHEMATICS) December, 2019 i Abstract We investigate asymptotically optimal multiple testing procedures for streams of sequential data in the context of prior information on the number of false null hypotheses (“signals”). We consider different pairs of error metrics (FDR/FNR, pFDR/pFNR, generalized error and k-family-wise error) and for each of these metrics, we consider sequential multiple testing procedures that achieve asymp- totic optimality in the sense of minimizing the expected sample size to first or- der as the type 1 and 2 versions of these error metrics approach 0 at arbitrary rates. This is done in the setting of testing simple null/alternative hypothe- sis pairs, and where there is some degree (e.g., zero) of prior information on the number of false null hypothesis (“signals”). In particular, we show that the “gap” and “gap-intersection” procedures, recently proposed and shown by Song and Fellouris (2017) to be asymptotically optimal for controlling type 1 and 2 family-wise error rates (FWEs), are also asymptotically optimal for controlling FDR/FNR when the critical values are appropriately adjusted. Generalizing this result, we show that these procedures, again with appropriately adjusted critical values, are asymptotically optimal for controlling any multiple testing error met- ric that is bounded between multiples of FWE in a certain sense. This enlarged class of metrics includes FDR/FNR but also pFDR/pFNR, the per comparison error rate, and other metrics. We also generalize the “sum" and “leap” proce- dures, proposed by Song and Fellouris (2016) for controlling generalized error and k-family-wise error with prior information on the number of signals. ii Acknowledgements I would like to express my sincere gratitude to my advisor Prof. Jay Bartroff for the continuous support of my Ph.D study and research. The completion of my dissertation would not have been possible without the support of my advisor. Besides my advisor, I would like to thank the rest of my thesis committee: Prof. Sergey Vladimir Lototsky and Prof. Jinchi Lv for their insightful comments and hard questions. Last but not the least, I would like to thank my family: my parents Xi- aomei Zhao, Sikun He and my husband Wei Zong for supporting me spiritually throughout my life. iii Contents Abstract i Acknowledgements ii Chapter 1 Introduction 1 1.1 Introduction and summary of this thesis 1.1.1 Prior information and Bayesian perspective 1.2 Background 1.3 Notation and assumptions Chapter 2 Procedures to control FDR, pFDR, and related metrics 13 2.1 Introduction 2.2 Set up and summary of our approach 2.3 Number of signals known exactly 2.3.1 Main result and its application to FDR/FNR and pFDR/pFNR control FDR/FNR control under known number of signals pFDR/pFNR control under known number of signals 2.3.2 Other multiple testing error metrics 2.4 Bounds on the number of signals 2.4.1 Main result and its application to FDR/FNR and pFDR/pFNR control iv FDR/FNR control under bounds on the number of signals pFDR/pFNR control under bounds on the number of sig- nals 2.4.2 Other multiple testing error metrics 2.5 Simulation study 2.6 Discussion Chapter 3 Procedures to control a generalized error metric 36 3.1 Introduction and Notation 3.2 Number of signals known exactly 3.2.1 Error control for the leap rule 3.2.2 Asymptotic optimality of the leap rule 3.3 Bounds on the number of signals 3.3.1 Error control for the leap sum rule 3.3.2 Asymptotic optimality for the leap sum rule 3.4 Effects of prior information Chapter 4 Procedures to control k-family-wise error rates 59 4.1 Introduction and Notation 4.2 Number of signals known exactly 4.2.1 Error control for the family-wise leap rule 4.2.2 Asymptotic optimality of the family-wise leap rule 4.3 Bounds on the number of signals 4.3.1 Error control for the family-wise leap sum rule 4.3.2 Asymptotic optimality of the family-wise leap sum rule 4.4 Effects of prior information Chapter 5 Discussion 83 v 5.1 Relationship to classification problems 5.2 Further extensions Bibliography 86 1 Chapter 1 Introduction 1.1 Introduction and summary of this thesis Multiple testing that is the simultaneous consideration of J 2 null hypothe- ses, is one of the oldest, yet still very active areas of statistical research. Recently, driven by applications where data is streaming or arrives sequentially, multi- ple testing procedures that can handle sequential data have been proposed and studied. Applications of this type include the analysis of streaming internet data (Wegman and Marchette, 2003), multiple channel signal detection in sensor net- works (Draglia, Tartakovsky, and Veeravalli, 1999; Mei, 2008), high throughput sequencing technology (Jiang and Salzman, 2012), and multi-arm and multiple endpoint clinical trials (Bartroff and Lai, 2010). In this thesis we will describe our research on sequential multiple hypotheses testing procedures which control an error metric (or metrics) at a prespecified level and are asymptotically optimal in the sense of minimizing the expected sample size to first order as that level approaches 0. This is done so with varying degrees of prior information on the number of false null hypotheses (“signals”), from knowing the exact number of signals (but not their identity), to bounds on the unknown number of signals, Chapter1. Introduction 2 which includes the case of the non-informative bounds[0, J], i.e., no prior infor- mation. Here the multiple testing error metrics are the widely-used False Dis- covery Rate (FDR) (Benjamini and Hochberg, 1995) and related metrics includ- ing the so-called Positive-FDR (pFDR) (Storey, 2002) and their type II analogs, generalized error (Song and Fellouris, 2016) which is similar to k-familywise er- ror rate but combines the type I and II versions, and family-wise type I and type II errors (Song and Fellouris, 2017). The rest of this thesis is organized as follows. In the remainder of Chapter 1 we describe relevant background in multiple se- quential testing and we we formulate the problem mathematically. In Chapter 2 we describe the procedures to control FDR, pFDR, and related metrics; most of the material from this chapter appears in He and Bartroff (2019), which is cur- rently under review at a journal and is posted on the ArXiv preprint server. In Chapter 3 we describe the procedures to control the generalized error metric. In Chapter 4 we describe the procedures to control family-wise error. In Chapter 5 we discuss some related topics and possible future directions. 1.1.1 Prior information and Bayesian perspective In the thesis, we consider J 2 independent data streams X j =fX j 1 , X j 2 , . . .g, j2[J], where [J] denotesf1, 2, . . . , Jg throughout. We use A, B, C, . . . [J] to denote the set of signals, i.e., the indices of the false null hypotheses, and thereforejAj is the number of signals in the data steam. If the number of signals is known to m then we will restrict our consideration to signal sets withjAj = m, and if the number of signals is only known to be bounded between two known values `, u then we will consider signal sets with ` jAj u. Thus the approach taken throughout this thesis is a quasi-Bayesian one. For example, by specifying Chapter1. Introduction 3 a known number m of signals, we are not specifying a single Bayesian prior distribution but rather a class of Bayesian priors – those with positive support on exactly m null hypotheses. Also, our objective, which is minimizing the expected sample size subject to bounds on the type 1 and 2 error metrics (or equivalently by a Lagrange multipliers argument, a linear combination of these terms), is not truly Bayesian since there is no expectation with respect to a specific prior distribution. One reason for our approach is that, even only considering simple hypotheses, prior distributions must take values on the 2 J points in the power set 2 [J] specification for each null, which may be quite difficult in practice even for statisticians, not to mention practitioners, especially when J is moderate or large. By considering classes of priors defined by the number of possible signals, we hope that this type of prior information may be more readily elicited from domain experts in applications, and the terms in our objective (average sample size and type 1 and 2 error metrics) more easily interpretable. 1.2 Background In the history of the development of multiple testing methodology, the early work was focused on the design of testing procedures that control the family- wise type I error in which the probability of at least one false positive is guar- anteed to be below a, an arbitrary chosen value in (0, 1). Letting V denote the number of false positives, then family-wise type I error is defined as FWE 1 = P(V 1). (1.1) The widely-used Bonferroni method (Bonferroni, 1936) for testing J null hy- potheses sets the rejection bound to be a/J for each hypothesis. Let H 1 , . . . , H J Chapter1. Introduction 4 be a family of hypotheses and p 1 , . . . , p J their corresponding p-values. We have FWE 1 = P ( J [ i=1 p i a J ) J å i=1 P p i a J = J a J = a. As can be seen from this calculation, this approach can be applied without any assumption on the joint distribution of the test statistics, however it also is fre- quently quite conservative and thus reduces the power of tests. Hochberg (1988) presented a step-down procedure by rejecting the k most significant hypotheses where k = max i : p i < a 1 J i+ 1 . The procedure accepts all hypotheses when the set is empty. And since Hochberg’s is based on the Simes test, so it holds only under non-negative dependence. More hypotheses will be rejected than the classical approach, since a/(J i+ 1) a/J. So for any p i < a/J, we have p i < a/(J i+ 1). If the num- ber of hypotheses is large, family-wise type I error can be too restrictive, since a/m decreases quickly with m and thus the number of rejected hypotheses may be small, if not zero, even in the presence of relatively small p-values. Ben- jamini and Hochberg (1995) proposed an alternative, and now ubiquitous, met- ric known as “False Discovery Rate", which is the expected value of the fraction of the rejected hypotheses that are true. Let R denote the number of rejected hypotheses, then FDR can be defined as : FDR= E V R . (1.2) Controlling FDR when the number of hypotheses is large allows the user to Chapter1. Introduction 5 maintain faith in the result in the sense that the fraction of false rejected hypothe- ses is small (at least in expectation) while often greatly increasing the number of hypotheses being rejected. Under the assumption that the J null hypotheses are independent, Benjamini and Hochberg (1995) also proposed a step-down proce- dure to control FDR by rejecting the k most significant hypotheses, where k = max i : p i < a i J . Here we give a simplistic example using Bonferroni’s method, Hochberg’s pro- cedure and Benjamini and Hochberg’s procedure: Suppose there are 4 null hy- potheses with associated p-values 0.01, 0.03, 0.06, 0.11 and a is 0.1. By us- ing Bonferroni’s method, we will only reject the first hypothesis, since 0.01 < 0.1/4< 0.03. By using Hochberg’s method, we will reject the first and the sec- ond hypotheses, since 0.11 > 0.1/(4 4+ 1) and 0.06 > 0.1/(4 3+ 1) and 0.03< 0.13/(4 2+ 1). By using Benjamini and Hochberg’s procedure, we re- ject the first three hypotheses, since 0.11> 0.1 4/4 and 0.06< 0.1 3/4. Thus we can see in this example each method rejects more null hypotheses than the last. All of the procedures mentioned above assume a fixed data set in the form of a set of J p-values. However, in many modern applications such as those men- tioned in 1.1, the data arrives over time and it is desirable to have the option of making accept/reject decisions before the next data point arrives. Sequential hy- pothesis testing, beginning with Wald’s (Wald, 1945) development during and after World War II, is a now well-developed field that we do not attempt to sum- marize in general here, because our focus is on procedures for multiple testing; for a wider summary of the development we refer the reader to Govindarajulu (1987), Siegmund (2010), or Tartakovsky, Nikiforov, and Basseville (2015). The Chapter1. Introduction 6 earliest proposed sequential hypotheses test for more than one null hypothesis is due to Sobel and Wald (1949) in deciding which of three simple hypothe- ses H 1 : q = d, H 2 : q = 0, or H 3 : q = d is true about a normal mean q, for a fixed value d > 0. Thus, this is essentially a 3-category classification problem. The Sobel-Wald test combines two sequential probability ratio tests (SPRTs) for different pairs of the three hypotheses, the comparison of H 1 versus H 3 being superfluous. For bivariate normal populations, Jennison and Turnbull (1993) proposed a sequential test of two one-sided hypotheses about the bivari- ate mean vector. A procedure for comparing three treatments was proposed by Siegmund (1993), and it is related to Paulson (1964)’s earlier procedure for selecting the largest mean of k normal distributions. These sequential multiple testing procedures described above are for very specific and stylized testing problems. Sequential procedures for more general data streams and hypotheses were not proposed until Bartroff and Lai (2010), who proposed a sequential test that controls the family-wise error of type I re- gardless of the structure of hypotheses. However, this procedure only controls family-wise type I error. De and Baron (2012) proposed synchronous (all the J hypotheses should stop sampling at the same time) sequential procedures that control both family-wise type I and type II errors and yields an optimal expected sampling cost under the regularity conditions. Bartroff and Song (2014) pro- posed an asynchronous (the J hypotheses can stop sampling at different times) procedure to control the family-wise error rates regardless of the between-stream correlation, and only requires arbitrary sequential test statistics that control the error rates for a given stream in isolation. And Bartroff and Song (2013) also proposed the sequential BH procedure that controls both FDR and false nondis- covery rate (FNR) under minimal assumptions about the data streams and only requires a test statistic for each data stream that controls the conventional type I Chapter1. Introduction 7 and II error probabilities. The proposed sequential procedures due Baron, Bartroff, De, and their coau- thors mentioned above were shown through simulation studies to offer substan- tial savings in the average sample size compared to the corresponding fixed- sample size tests in some commonly-encountered testing situations. However, due to the generality of the assumptions, any notion of optimality (even asymp- totic) was not addressed. Traditionally, optimality in the sequential setup refers to a procedure with guaranteed control of some error metric while the proce- dure’s expected sample size is minimized. And asymptotic optimality usually means that this expected sample size is minimized as the bound on the error metric (or metrics) approaches zero. Song and Fellouris (2016) were the first to obtain asymptotic optimality results in a somewhat general setting – although confined to simple-vs.-simple tests and independent data streams – while con- trolling the classical family-wise type I and type II errors with prior information on the number of signals and proved that the procedure is asymptotically op- timal when the family-wise error rates go to 0. They called their procedures “gap” and “intersection” rules because, when the number of signals is known to be m, the procedure stops sampling as soon as the gap c of the log likelihood ratio between the m th and (m+ 1) th becomes large enough. And when the number of signals is unknown, the procedure stops sampling as soon as all the log likelihood ratios are out of the range(a, b). If a null hypothesis’ log like- lihood ratio is larger than b, we reject the hypothesis and if the log likelihood ratio is smaller than a, we accept the hypothesis. Chapter1. Introduction 8 1.3 Notation and assumptions In this section we introduce the notation and assumptions used throughout the chapters that follow. Other specific notation and assumptions will be intro- duced in those chapters as needed. We consider J 2 independent data streams X j =fX j 1 , X j 2 , . . .g, j2[J], (1.3) recalling that [J] = f1, 2, . . . , Jg. We assume throughout the thesis that the streams are independent of each other, but not always that the elements of a given stream X j 1 , X j 2 , . . . are independent. In particular, we will prove error con- trol of the proposed procedures only under independence of the streams, but make the additional assumption that each stream is made up of i.i.d. observa- tions in order to prove asymptotic optimality. Letting P j denote the probability distribution of X j , we consider simultane- ously testing J null-alternative simple hypothesis pairs H j 0 : P j = P j 0 vs. H j 0 : P j = P j 1 , j2[J], (1.4) where, for each j, the P j 0 and P j 1 are known and distinct distributions, assumed to be mutually absolutely continuous. We utilize the informal but illustrative terminology that refers to the jth stream as noise (resp. signal) if H j 0 (resp. H j 1 ) is true. We shall refer to the subscript n in X j n as time and assume that at time n, the observations X j 1 , X j 2 , . . . , X j n , j2[J], (and only these observations) have been observed. More precisely, let s n denote the s-field generated by the observations available at time n, i.e., s n = s(fX j i : i 2 [n], j 2 [J]g). We define a sequential test (or procedure) of (1.4) as a pair Chapter1. Introduction 9 of random variables (T, D) such that T is afs n g-stopping time (i.e., the event fT = ng2 s n ) and D = (D 1 , . . . , D J ) is a s T -measurable decision rule taking values inf0, 1g J . The interpretation of (T, D) is that sampling of all streams is terminated at time T and the procedure classifies the jth stream as noise (resp. signal) if D j = 0 (resp. D j = 1), for each j2[J]. For each pair of distributions P j 0 and P j 1 in (1.4), let f j 0 and f j 1 denote the re- spective density functions with respect to some common measure m j . Denote the log-likelihood ratio for the jth stream at time n by l j (n) := log f j 1 (X j 1 , . . . , X j n ) f j 0 (X j 1 , . . . , X j n ) and the Kullback-Leibler information numbers by I j 0 := Z logl j (1) 1 f j 0 (X j 1 )dm j and I j 1 := Z logl j (1) f j 1 (X j 1 )dm j . We assume the likelihood ratio statistics satisfy P j 0 lim n!¥ l j (n)=¥ = P j 1 lim n!¥ l j (n)=¥ = 1 (1.5) and for each j2[J] we will assume that there are information numbers I j 0 and I j 1 such that the following Strong Law of Large Numbers (SLLN) holds: P j 1 lim n!¥ l j (n) n = I j 1 = P j 0 lim n!¥ l j (n) n =I j 0 = 1 (1.6) for all j2[J]. For example, if the elements of the stream X j 1 , X j 2 , . . . are i.i.d., then (1.6) follows from the strong law of large numbers. Since, as mentioned in the first paragraph of this section, we assume the streams are independent through- out but only make the i.i.d. assumption for some of our results, the assumption Chapter1. Introduction 10 (1.6) guarantees that the proposed sequential tests will terminate a.s. even when the i.i.d. assumption is not made. In what follows we will make frequent use of the log-likelihood ratios’ order statistics which we denote by l (1) (n) l (2) (n) . . . l (J) (n), (1.7) with ties broken arbitrarily. To refer to a particular order statistic let i j (n), j2[J], be such thatl i j (n) (n)= l (j) (n). Also, lettingjj denote set cardinality through- out, define p(n) := fj2[J] : l j (n)> 0g to be the number of positive l j at time n. We will describe states of nature by subsets A[J] which we call signal sets, describing the situation where H j 0 is false (signal) for all j2 A, and H j 0 is true (noise) for all j2 A c := [J]n A. For a signal set A, let P A denote the probability measure determined by A, i.e., P A := J O j=1 P j 1fj2Ag , (1.8) where 1fg denotes an indicator function throughout. Let E A denote expectation with respect to P A . For a given signal set A, let h A 0 := min j2A c I j 0 and h A 1 := min j2A I j 1 , which can be thought of as the “worst case” information numbers for signal set A, and appear in the expected sample sizes of optimal procedures. Chapter1. Introduction 11 For any two subsets A, C [J] we denote by l A,C the log-likelihood ratio process of P A versus P C , i.e., l A,C := å k2AnC l k (n) å k2CnA l k (n). (1.9) And we define information number difference between the two subsets A, C I A,C := å k2AnC I k 1 + å k2CnA I k 0 . (1.10) Above in (1.1) and (1.2) we defined FWE 1 and FDR with respect to an arbi- trary distribution. To define these in terms of a specific signal set A we extend our notation as follows: For any multiple testing procedure under consideration, let V denote the number of true null hypotheses rejected (i.e., the number of false positives), W the number of false null hypotheses accepted (i.e., the number of false nega- tives), and R the number of null hypotheses rejected. The number of null hy- potheses accepted is therefore J R. Under signal set A the type 1 and 2 family- wise error rates (FWEs) are FWE 1,A = P A (V 1), FWE 2,A = P A (W 1) and the false discovery and non-discovery rates (FDR, FNR) are FDR A = E A V R_ 1 , FNR A = E A W (J R)_ 1 , where x_ y= maxfx, yg. In these and other multiple testing metrics, we will in- clude the procedure being evaluated as an argument (e.g., FWE 1,A (T, D)) when needed but omit it when it is clear from the context or when a statement holds Chapter1. Introduction 12 for any procedure. Likewise, we will omit the subscript A in expressions that hold for arbitrary signal sets, such as in the next chapters. 13 Chapter 2 Procedures to control FDR, pFDR, and related metrics 2.1 Introduction In this chapter we describe the procedures to control FDR, pFDR, and related metrics; most of the material from this chapter appears in He and Bartroff (2019). After introducing notation and describing our approach in Section 2.2, the case of the number of signals known exactly is addressed in Section 2.3 where first the general result for arbitrary error metrics is stated, followed by its application to FDR/FNR and pFDR/pFNR. The case of bounds on the number of signals is ad- dressed in Section 2.4 where, again, the general result for arbitrary error metrics followed by its application to FDR/FNR and pFDR/pFNR. Simulation studies of procedures for FDR/FNR control in finite-sample settings are presented in Section 2.5, and we conclude with a discussion of related issues in Section 2.6. 2.2 Set up and summary of our approach All the notation and assumptions in this chapter is the same as in Section 1.3. For the asymptotic theory governing optimal FDR and FNR control it will suf- fice to consider the following elementary bounds between these metrics and the Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 14 type 1 and 2 FWEs. On one hand, FDR= E V R_ 1 = E V R_ 1 1fV 1g E(1 1fV 1g)= P(V 1)= FWE 1 . (2.1) On the other hand, FDR= E V R_ 1 1fV 1g E 1 J 1fV 1g = 1 J P(V 1)= 1 J FWE 1 . (2.2) Similar arguments show that 1 J FWE 2 FNR FWE 2 . (2.3) Although crude, it will turn out that bounds like (2.1)-(2.3) suffice to show that sequential procedures that are asymptotically optimal for FWE control are also asymptotically optimal for FDR/FNR control. More generally, we will show that the sequential procedures that are asymptotically optimal for FWE control are also asymptotically optimal (with slightly modified critical values) for control of any multiple testing error metric whose type 1 and 2 versions are bounded be- low some constant multiple of the corresponding FWE i when evaluated on the optimal procedure, and are bounded above some constant multiple of the cor- responding FWE i when evaluated on any procedure; these statements are made precise in conditions (i)-(ii) of Theorems 2.3.1 and 2.4.1. To state these more gen- eral results, we denote the type 1 and 2 versions of a generic multiple testing er- ror by MTE = (MTE 1 , MTE 2 ), which is any pair of functions mapping multiple testing procedures into[0, 1]. As above, we will add a signal set A as a subscript Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 15 and a procedure (T, D) as an argument when needed. Our more general re- sults in Theorems 2.3.1 and 2.4.1 will produce asymptotic optimality results for FDR/FNR in Corollaries 2.3.1 and 2.4.1 and for pFDR/pFNR in Corollaries 2.3.2 and 2.4.2, after verifying that pFDR/pFNR satisfy similar bounds. We will useP to denote a signal class which is a collection of signal sets; more precisely,P is an element of the power set of[J]. We will consider the classes of sequential tests(T, D) controlling the type 1 and 2 versions of various multiple testing error metrics at specific levels 1 a,b> 0. For a generic metric MTE, let D MTE P (a,b)=f(T, D) : MTE 1,A a and MTE 2,A b for all A2Pg. (2.4) When the MTE is FWE we have D FWE P (a,b)=f(T, D) : FWE 1,A a and FWE 2,A b for all A2Pg and, by a slight but obvious abuse of notation, for FDR/FNR control we have D FDR P (a,b)=f(T, D) : FDR A a and FNR A b for all A2Pg. 2.3 Number of signals known exactly For m2 [J 1] letP m =fA [J] :jAj = mg. ThusP m is the class of signal sets with exactly m signals. For a given such m, Song and Fellouris (2017) defined the following gap rule(T G (c), D G (c)) with threshold c> 0, which is the sequential procedure that stops sampling as soon as the “gap” between the mth 1 Here we do not explicitly require that a,b < 1 in the definitions of the classes that follow. This is because below we will multiply and divide inputs a,b to the class (2.4) by various pos- itive constants. This does not present a problem because, for example, if a 1 then the class requirement inD MTE P (a,b) that MTE 1,A a will automatically be satisfied since MTE i 2[0, 1] by assumption. Similar statements apply if b 1. Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 16 and (m+ 1)st ordered log-likelihood ratio statistics is at least c. The decision taken at that time is that the null hypotheses with the largest m statistics contain signal. That is, T G (c) := inffn 1 : l (m) (n)l (m+1) (n) cg, (2.5) D G (c) :=fi 1 (T G (c)), ..., i m (T G (c))g. (2.6) Theorem 2.3.1 establishes admissability and asymptotic optimality of the gap rule for control of any metric whose type 1 version is is bounded above by some constant multiple of FWE 1 when evaluated on the gap rule in the sense of (2.7), and whose type 2 version is is bounded above by some constant multiple of FWE 2 when evaluated on any procedure in the sense of (2.8). 2.3.1 Main result and its application to FDR/FNR and pFDR/pFNR control Theorem 2.3.1. Fix m 2 [J 1] and let (T G (c), D G (c)) denote the gap rule with number of signals m and threshold c > 0. Let MTE be multiple testing error metric such that: (i) there is a constant C 1 <¥ such that MTE i,A (T G (c), D G (c)) C 1 FWE i,A (T G (c), D G (c)) (2.7) for i = 1 and 2, for all A2P m , and for all c> 0, and (ii) there is a constant C 2 > 0 such that MTE i,A (T, D) C 2 FWE i,A (T, D) (2.8) for i = 1 and 2, for all A2P m , and for all procedures(T, D). Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 17 Given a,b2 (0, 1) let (T G , D G ) denote the gap rule with number of signals m and threshold c=j log((a/C 1 )^(b/C 1 ))j+ log(m(J m)). Then the following hold. 1. Under the assumption that the streams X 1 , . . . , X J are independent,(T G , D G ) is admissible for MTE control. That is, (T G , D G )2D MTE P m (a,b). (2.9) 2. Under the additional assumption that each stream X j = (X j 1 , X j 2 , . . .) is made up of i.i.d. random variables X j 1 , X j 2 , . . ., the procedure (T G , D G ) is asymptotically optimal for MTE control with respect to classP m . That is, for all A2P m , E A (T G ) j log(a^b)j h A 1 +h A 0 inf (T,D)2D MTE P m (a,b) E A (T) asa,b! 0. Proof. For part 1, fix arbitrary A2P m . Song and Fellouris (2017, Theorem 3.1) establish that FWE 1,A (T G , D G ) a/C 1 and FWE 2,A (T G , D G ) b/C 1 . Applying (2.7) yields MTE 1,A (T G , D G ) C 1 FWE 1,A (T G , D G ) C 1 (a/C 1 )= a. A similar argument shows that MTE 2,A (T G , D G ) b, establishing (2.9). Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 18 For part 2, fix arbitrary A2P m and consider a,b! 0. Letting k denote j log(a^b)j!¥, j log((a/C 1 )^(b/C 1 ))j= k+ log C 1 = k+ O(1). (2.10) The first inequality in the following is established by Song and Fellouris (2017, Lemma 5.2), and the rest use (2.10). We have E A (T G ) j log((a/C 1 )^(b/C 1 ))j h A 1 +h A 0 + O q j log((a/C 1 )^(b/C 1 ))j = k+ O(1) h A 1 +h A 0 + O q k+ O(1) = k h A 1 +h A 0 + O p k = k h A 1 +h A 0 (1+ o(1)). (2.11) To establish that the last expression in (2.11) is also a lower bound for any pro- cedure inD MTE P m (a,b), it follows from (2.8) that D MTE P m (a,b)D FWE P m (a/C 2 ,b/C 2 ), thus inf (T,D)2D MTE P m (a,b) E A (T) inf (T,D)2D FWE P m (a/C 2 ,b/C 2 ) E A (T). Song and Fellouris (2017, Theorem 5.3) establish that the latter is j log((a/C 2 )^(b/C 2 ))j h A 1 +h A 0 (1+ o(1)) k h A 1 +h A 0 by arguments similar to (2.11). Thus inf (T,D)2D MTE P m (a,b) E A (T) k h A 1 +h A 0 (1+ o(1)) Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 19 and combining this with (2.11) gives the desired result. FDR/FNR control under known number of signals To apply Theorem 2.3.1 to FDR/FNR, we see that the upper bounds on FDR/FNR in (2.1) and (2.3) suffice 2 for condition (i) of the theorem with C 1 = 1, and the lower bounds in (2.2) and (2.3) suffice for condition (ii) with C 2 = 1/J, yielding the following corollary. Corollary 2.3.1. For m2 [J 1] and a,b2 (0, 1), let(T G , D G ) denote the gap rule with number of signals m and threshold c=j log(a^b)j+ log(m(J m)). 1. Under the assumption that the streams X 1 , . . . , X J are independent,(T G , D G ) is admissible for FDR/FNR control. That is, (T G , D G )2D FDR P m (a,b). 2. Under the additional assumption that each stream X j = (X j 1 , X j 2 , . . .) is made up of i.i.d. random variables X j 1 , X j 2 , . . ., the procedure (T G , D G ) is asymptotically optimal for FDR/FNR control with respect to classP m . That is, for all A2P m , E A (T G ) j log(a^b)j h A 1 +h A 0 inf (T,D)2D FDR P m (a,b) E A (T) asa,b! 0. pFDR/pFNR control under known number of signals Continuing to use the notation, pFDR and its type 2 analog pFNR are defined as pFDR= E V R R 1 and pFNR= E W J R J R 1 . (2.12) 2 In fact, (2.1) and (2.3) are stronger than what is needed since they hold for all procedures, whereas condition (i) just requires this to hold for the gap rule. Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 20 As above, we will add the signal set A in the subscript and a procedure as an argument when needed. Fora,b> 0 and signal classP, define D pFDR P (a,b)=f(T, D) : pFDR A a and pFNR A b for all A2Pg. To apply Theorem 2.3.1 to pFDR/pFNR, unlike with FDR/FNR it is not pos- sible to provide universal upper bounds like (2.1) and (2.3). Proceeding simi- larly, one obtains pFDR= E V R R 1 = E V R R 1, V 1 P(V 1jR 1) 1 P(V 1) P(R 1) = FWE 1 P(R 1) . (2.13) Similarly, pFNR P(W 1)/P(J R 1), and because P(R 1) and P(J R 1) cannot be bounded below in general, universal upper bounds like (2.1) and (2.3) do not hold for pFDR/pFNR. However, condition (i) in Theorem 2.3.1 merely requires such upper bounds to hold for the gap rule (T G , D G ) with m signals, which always rejects exactly m nulls, and recalling that 1 m J 1 by assumption (i.e., m2[J 1]), the gap rule satisfies P(R 1)= P(J R 1)= 1. Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 21 Thus, (2.13) yields pFDR(T G , D G ) FWE 1 and pFNR(T G , D G ) FWE 2 , so con- dition (i) of the theorem holds with C 1 = 1. The lower bounds are more straight- forward: pFDR= E V R R 1, V 1 P(V 1jR 1) 1 J P(V 1) P(R 1) 1 J FWE 1 1 = 1 J FWE 1 , (2.14) and similarly pFNR(1/J) FWE 2 . (2.15) Thus, condition (ii) of the theorem is satisfied with C 2 = 1/J, yielding the fol- lowing corollary. Corollary 2.3.2. For m2 [J 1] and a,b2 (0, 1), let(T G , D G ) denote the gap rule with number of signals m and threshold c=j log(a^b)j+ log(m(J m)). 1. Under the assumption that the streams X 1 , . . . , X J are independent,(T G , D G ) is admissible for pFDR/pFNR control. That is, (T G , D G )2D pFDR P m (a,b). 2. Under the additional assumption that each stream X j = (X j 1 , X j 2 , . . .) is made up of i.i.d. random variables X j 1 , X j 2 , . . ., the procedure (T G , D G ) is asymptotically optimal for pFDR/pFNR control with respect to classP m . That is, for all A2P m , E A (T G ) j log(a^b)j h A 1 +h A 0 inf (T,D)2D pFDR P m (a,b) E A (T) asa,b! 0. Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 22 2.3.2 Other multiple testing error metrics Theorem 2.3.1 applies to any multiple testing error metrics which satisfy the theorem’s conditions (i) and (ii). In addition to FDR/FNR and pFDR/pFNR these include the per-comparison error rate (PCER) E(V/J) as defined by Ben- jamini and Hochberg (1995), the false positive rate E(V/m) (e.g., Burke et al., 1988), per-family error rate 3 (PFER) EV which appears in social and behavioral sci- ence research e.g., Frane, 2015; Keselman, 2015, and their type 2 analogs. These type 1 metrics are all proportional to EV and for the gap rule, EV = E(V1fV 1g) m P(V 1)= m FWE 1 , (2.16) and for any procedure, EV = E(V1fV 1g) 1 P(V 1)= FWE 1 . (2.17) Similar statements apply to the type 2 versions. 2.4 Bounds on the number of signals In this section we consider asymptotically optimal procedures for scenar- ios in which the number of signals is known to be between two given val- ues 0 ` < u J, the ` = u case having already been considered in Sec- tion 2.3. That is, we consider asymptotic optimality in signal classes of the form P `,u :=fA [J] : `jAj ug for such`, u. Mirroring our results for when the number of signals is known exactly, we show that the sequential procedures that are asymptotically optimal for FWE control are also asymptotically optimal 3 This metric is not constrained to take values in [0, 1] as we assumed above, but the theory there can be modified to allow metrics to take any nonnegative values with only notational changes. Alternatively, one could think of standardizing the false positive rate by dividing by m or J, leading to the other two metrics mentioned. Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 23 (with modified critical values) for control of any multiple testing error metric whose type 1 and 2 versions are bounded between constant multiples of the corresponding FWE i in certain situations, made precise in conditions (i)-(ii) of Theorem 2.4.1. Given such bounds` and u, Song and Fellouris (2017) defined the following gap-intersection rule (T GI , D GI ), depending on four positive thresholds a, b, c, d, in terms of the stopping times t 1 := inffn 1 : l (`+1) (n)a, l (`) (n)l (`+1) (n) cg, (2.18) t 2 := inffn 1 : ` p(n) u and l (j) (n) / 2(a, b) for all j2[J]g, (2.19) t 3 := inffn 1 : l (u) (n) b, l (u) (n)l (u+1) (n) dg. (2.20) In (2.18)-(2.20) we set l (0) (n) =¥ and l (J) (n) = ¥ for all n to handle the cases`= 0 or u= J. Finally, the gap-intersection rule(T GI , D GI ) is defined as T GI := minft 1 ,t 2 ,t 3 g and D GI :=fi 1 (T GI ), ..., i p 0(T GI )g, where p 0 := (p(T GI )_`)^ u is the value inf`,`+ 1, . . . , ug closest to p(T GI ). Song and Fellouris (2017) describe t 2 as similar to De and Baron’s (2012) “in- tersection rule,” which requires only the second condition in (2.19), but modi- fied to incorporate the prior information on the number of signals by requiring ` p(t 2 ) u. The stopping times t 1 and t 3 provide needed additional effi- ciency when the number of signals equals` or u. Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 24 2.4.1 Main result and its application to FDR/FNR and pFDR/pFNR control Theorem 2.4.1. Fix integers 0 ` < u J and let (T GI , D GI ) denote the gap- intersection rule with bounds`, u on the number of signals and thresholds a, b, c, d> 0. Let MTE be multiple testing error metric such that: (i) there is a constant C 1 such that MTE i,A (T GI , D GI ) C 1 FWE i,A (T GI , D GI ) (2.21) for i = 1 and 2, for all A2P `,u , and for all thresholds a, b, c, d> 0, and (ii) there is a constant C 2 such that MTE i,A (T, D) C 2 FWE i,A (T, D) (2.22) for i = 1 and 2, for all A2P `,u , and for all procedures(T, D). Given a,b2 (0, 1) let(T GI , D GI ) denote the gap rule with bounds`, u on the number of signals, and thresholds a=j log(b/C 1 )j+ log J, b=j log(a/C 1 )j+ log J, c=j log(a/C 1 )j+ log((J`)J), d=j log(b/C 1 )j+ log(uJ). Then the following hold. 1. Under the assumption that the streams X 1 , . . . , X J are independent, (T GI , D GI ) is admissible for MTE control. That is, (T GI , D GI )2D MTE P `,u (a,b). (2.23) Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 25 2. Under the additional assumption that each stream X j = (X j 1 , X j 2 , . . .) is made up of i.i.d. random variables X j 1 , X j 2 , . . ., the procedure(T GI , D GI ) is asymptotically optimal for MTE control with respect to classP `,u . That is, for all A2P `,u , E A (T GI ) inf (T,D)2D MTE P `,u (a,b) E A (T) 8 > > > > > > < > > > > > > : maxfj logbj/h A 0 ,j logaj/(h A 0 +h A 1 )g, ifjAj=` maxfj logbj/h A 0 ,j logaj/h A 1 g, if`<jAj< u maxfj logaj/h A 1 ,j logbj/(h A 0 +h A 1 )g, ifjAj= u (2.24) asa,b! 0. Proof. For part 1, fix arbitrary A2P `,u . Song and Fellouris (2017, Theorem 3.2) establish that FWE 1,A (T GI , D GI ) a/C 1 and FWE 2,A (T GI , D GI ) b/C 1 . Applying (2.21) yields MTE 1,A (T GI , D GI ) C 1 FWE 1,A (T GI , D GI ) C 1 (a/C 1 )= a, and a similar argument shows that MTE 2,A (T GI , D GI ) b, establishing (2.23). For part 2, fix arbitrary A2P `,u and consider a,b! 0. Lettingk(a,b)!¥ denote the corresponding expression in (2.24), it is not hard to see that k(a/C 1 ,b/C 1 )= k(a,b)+ O(1), (2.25) Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 26 which can be verified for each of the three cases. For example, for the first case, k(a/C 1 ,b/C 1 )= maxfj log(b/C 1 )j/h A 0 ,j log(a/C 1 )j/(h A 0 +h A 1 )g = maxfj logbj/h A 0 + O(1),j logaj/(h A 0 +h A 1 )+ O(1)g = maxfj logbj/h A 0 ,j logaj/(h A 0 +h A 1 )g+ O(1) = k(a,b)+ O(1), with the other cases being similar. Song and Fellouris (2017, Theorem 5.8) estab- lish that E A (T GI ) k(a/C 1 ,b/C 1 ) k(a,b), (2.26) the second equivalence by (2.25). To establish that this last is also an asymptotic lower bound inD MTE P `,u (a,b), it follows from (2.22) thatD MTE P `,u (a,b)D FWE P `,u (a/C 2 ,b/C 2 ), thus inf (T,D)2D MTE P `,u (a,b) E A (T) inf (T,D)2D FWE P `,u (a/C 2 ,b/C 2 ) E A (T). Song and Fellouris (2017, Theorem 5.8) establish that the latter is equivalent to k(a/C 2 ,b/C 2 ) k(a,b), again by (2.25). Thus inf (T,D)2D MTE P `,u (a,b) E A (T) k(a,b)(1+ o(1)) and combining this with (2.26) gives the desired result. FDR/FNR control under bounds on the number of signals The application of Theorem 2.4.1 to FDR/FNR is again immediate from the bounds (2.1)-(2.3), from which we see that condition (i) of the theorem holds with C 1 = 1 and condition (ii) holds with C 2 = 1/J, yielding the following corollary. Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 27 Corollary 2.4.1. Given a,b2 (0, 1) and integers 0`< u J, let(T GI , D GI ) de- note the gap-intersection rule with bounds`, u on the number of signals, and thresholds a=j logbj+ log J, b=j logaj+ log J, c=j logaj+ log((J`)J), d=j logbj+ log(uJ). Then the following hold. 1. Under the assumption that the streams X 1 , . . . , X J are independent, (T GI , D GI ) is admissible for FDR/FNR control. That is, (T GI , D GI )2D FDR P `,u (a,b). 2. Under the additional assumption that each stream X j = (X j 1 , X j 2 , . . .) is made up of i.i.d. random variables X j 1 , X j 2 , . . ., the procedure(T GI , D GI ) is asymptotically optimal for FDR/FNR control with respect to classP `,u . That is, for all A2P `,u , E A (T GI ) inf (T,D)2D FDR P `,u (a,b) E A (T) 8 > > > > > > < > > > > > > : maxfj logbj/h A 0 ,j logaj/(h A 0 +h A 1 )g, ifjAj=` maxfj logbj/h A 0 ,j logaj/h A 1 g, if`<jAj< u maxfj logaj/h A 1 ,j logbj/(h A 0 +h A 1 )g, ifjAj= u asa,b! 0. pFDR/pFNR control under bounds on the number of signals In applying Theorem 2.4.1 to pFDR/pFNR, some care must be taken when considering condition (i) of the theorem since pFDR and pFNR cannot be univer- sally bounded above by a constant multiple of FWE 1 and FWE 2 , respectively, as Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 28 in (2.13). Although condition (i) only requires such bounds to hold for the gap- intersection rule, these may still fail when the lower bound is` = 0 and P(R 1) is close to zero, or when the upper bound is u= J and P(J R 1) is close to zero. For this reason, in the following corollary which applies Theorem 2.4.1 to pFDR/pFNR control, we restrict attention to prior bounds` 1 and u J 1, under which the gap-intersection rule satisfies P(R = 0) = P(R = J) = 0 and thus condition (i) holds with C 1 = 1 by (2.13). We note that problems with pFDR and pFNR when P(R 1) and P(J R 1), respectively, are close to zero are not unique to our setup here, and these quantities are of course undefined when these probabilities equal zero; see Section 2.6 for more discussion of this topic. The needed lower bounds in condition (ii) of the theorem are again provided by the universal bounds (2.14)-(2.15) with C 2 = 1/J. Corollary 2.4.2. Given a,b2 (0, 1) and integers 1`< u J 1, let(T GI , D GI ) denote the gap-intersection rule with bounds`, u on the number of signals, and thresh- olds a=j logbj+ log J, b=j logaj+ log J, c=j logaj+ log((J`)J), d=j logbj+ log(uJ). Then the following hold. 1. Under the assumption that the streams X 1 , . . . , X J are independent, (T GI , D GI ) is admissible for pFDR/pFNR control. That is, (T GI , D GI )2D pFDR P `,u (a,b). 2. Under the additional assumption that each stream X j = (X j 1 , X j 2 , . . .) is made up of i.i.d. random variables X j 1 , X j 2 , . . ., the procedure(T GI , D GI ) is asymptotically Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 29 optimal for pFDR/pFNR control with respect to classP `,u . That is, for all A2 P `,u , E A (T GI ) inf (T,D)2D pFDR P `,u (a,b) E A (T) 8 > > > > > > < > > > > > > : maxfj logbj/h A 0 ,j logaj/(h A 0 +h A 1 )g, ifjAj=` maxfj logbj/h A 0 ,j logaj/h A 1 g, if`<jAj< u maxfj logaj/h A 1 ,j logbj/(h A 0 +h A 1 )g, ifjAj= u asa,b! 0. 2.4.2 Other multiple testing error metrics In addition to FDR/FNR and pFDR/pFNR, Theorem 2.4.1 applies to any multiple testing error metrics which satisfy the theorem’s conditions including the metrics mentioned in Section 2.3.2. The argument is similar, with the upper bound (2.16) being replaced by u FWE 1 for the gap-intersection rule. 2.5 Simulation study In this section we study the non-asymptotic performance of the sequential gap rule for FDR/FNR control as well as comparable fixed-sample procedures based on the Benjamini and Hochberg (1995) procedure (BH) through a simula- tion study in the setting of Section 2.3 where the number m of signals is known exactly. We consider J independent streams X j 1 , X j 2 , . . ., j2 [J], of i.i.d. N(m, 1) data with hypotheses H j 0 : m = 0 vs. H j 1 : m = 1/2, j2 [J]. Table 2.1 contains the operating characteristics of three procedures under this setup with J = 10 and all possible values of m2 [J 1] in the table’s rows. The columns under the heading Gap Rule in Table 2.1 describe the performance of the gap rule as defined in (2.5)-(2.6) with values of the threshold c listed in the table. In order to Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 30 study the non-asymptotic performance of this procedure, the asymptotic value of c in Corollary 2.3.1 was dispensed with and for each m, a value of c was deter- mined by Monte Carlo simulation such that both the achieved FDR and FNR are close to, but no larger than, the nominal values a = b = .05. We note that this same approach is available to users in practice since m and the simple hypothe- ses H j 0 , H j 1 are all known and thus the achieved FDR and FNR can be estimated to arbitrary accuracy via Monte Carlo before gathering any data. For the gap rule with c determined in this way, the expected stopping time ET and achieved FDR and FNR, estimated from 10,000 Monte Carlo replications, are given in the table with their standard errors given in parentheses. A natural competitor for the gap rule is the fixed-sample BH procedure. In implementing the BH procedure two parameters must be chosen: the proce- dure’s nominal FDR control level a and its fixed sample size, denoted by n in Table 2.1. In order to have an even-handed comparison, the same nominal level a = .05 as the gap rule was used, and for each m the BH procedure’s sample size n was then chosen so that its achieved FNR was as close as possible to that of the gap rule. The columns under the heading BH in Table 2.1 contain the op- erating characteristics of the BH procedure with parameters chosen in this way, where again the FDR and FNR are estimated from 10,000 Monte Carlo repli- cations with standard errors given in parentheses. The first of these columns contains the fixed sample size n as well as the expected savings 1 ET/n in sample size of the sequential gap rule over the fixed-sample rule. Although the BH procedure as implemented in the previous paragraph does make some use of knowledge of the number m of signals through the choice of its fixed sample size n for each m, the decision rule itself does not make explicit use of m and may not necessarily reject exactly m null hypotheses. In order compare with a fixed-sample procedure that does make explicit use of m, like Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 31 TABLE 2.1: Operating characteristics of sequential gap rule and fixed-sample BH and BH m procedures in simu- lation study as described in Section 2.5 with J = 10 data streams. Gap Rule BH BH m m c ET FDR FNR n (Savings) FDR FNR n (Savings) FDR FNR 1 3.5 29.0 (0.15) 4.30 (0.20) 0.48 (0.02) 70 (59%) 4.49 (0.15) 0.61 (0.02) 50 (42%) 4.54 (0.21) 0.50 (0.02) 2 2.9 31.6 (0.14) 4.60 (0.15) 1.15 (0.04) 60 (47%) 3.97 (0.12) 1.50 (0.04) 46 (31%) 4.74 (0.15) 1.18 (0.04) 3 2.6 31.7 (0.14) 4.75 (0.12) 2.04 (0.05) 59 (46%) 3.55 (0.09) 2.05 (0.05) 45 (30%) 4.40 (0.11) 1.88 (0.05) 4 2.3 30.2 (0.13) 4.59 (0.10) 3.06 (0.07) 54 (44%) 2.84 (0.08) 3.24 (0.07) 40 (25%) 4.80 (0.10) 3.20 (0.06) 5 2.1 28.7 (0.12) 4.66 (0.09) 4.66 (0.09) 52 (45%) 2.55 (0.07) 4.67 (0.08) 37 (22%) 4.75 (0.09) 4.75 (0.09) 6 2.3 30.5 (0.13) 3.18 (0.07) 4.77 (0.10) 54 (44%) 2.06 (0.05) 4.53 (0.09) 40 (24%) 3.32 (0.07) 4.99 (0.10) 7 2.5 30.8 (0.13) 2.14 (0.05) 4.90 (0.12) 56 (45%) 1.50 (0.04) 4.91 (0.11) 43 (28%) 2.10 (0.05) 4.90 (0.12) 8 2.8 30.7 (0.14) 1.22 (0.04) 4.89 (0.15) 60 (49%) 0.96 (0.03) 4.92 (0.13) 45 (32%) 1.31 (0.04) 5.24 (0.15) 9 3.4 28.5 (0.15) 0.49 (0.02) 4.39 (0.20) 65 (56%) 0.50 (0.02) 4.77 (0.15) 50 (43%) 0.48 (0.02) 4.33 (0.20) Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 32 the gap rule, we also implemented a fixed-sample procedure (denoted BH m ) that always rejects exactly m null hypotheses by sampling n observations from each stream and rejecting the m null hypotheses with the smallest p-values. Such a procedure makes no use of a nominal level a, so we implemented BH m by choosing n so that its achieved FDR and FNR are close to, but no larger than, the nominal levels a = b = .05, similar to how the gap rule’s threshold c was determined. The last three columns of Table 2.1, under the heading BH m , give the operating characteristics of this procedure, including the savings in sample size achieved by the gap rule. From the Savings columns in Table 2.1 we see that the sequential gap rule provides dramatic efficiency gains, in terms of average sample size, relative to the BH and BH m procedures in this setting. The savings is in the 40%-60% range versus BH, and less so but still substantial in the 20%-45% range versus BH m , which makes more explicit use of the true number of signals m than the BH procedure. The savings in sample size is largest for values of m near 0 and J, and decreases for m near J/2. The achieved FDR and FNR values are comparable among the three procedures. Table 2.2 contains the operating characteristics of these procedures with pa- rameters chosen in the analogous way for the J = 100 data stream version of the same setup. The three procedures have a similar relationship to each other as in the J = 10 setting, however the savings in sample size is less pronounced for this larger value of J. 2.6 Discussion Summary and extensions We have shown that sequential procedures proposed and shown to be asymp- totically optimal by Song and Fellouris (2017) for FWE control can be made by Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 33 TABLE 2.2: Operating characteristics of sequential gap rule and fixed-sample BH and BH m procedures in simu- lation study as described in Section 2.5 with J = 100 data streams. Gap Rule BH BH m m c ET FDR FNR n (Savings) FDR FNR n (Savings) FDR FNR 1 3.9 48.8 (0.21) 4.43 (0.21) 0.04 (0.00) 90 (46%) 4.77 (0.16) 0.08 (0.00) 77 (37%) 4.74 (0.21) 0.05 (0.00) 10 1.9 61.0 (0.16) 4.65 (0.06) 0.52 (0.01) 70 (13%) 4.52 (0.06) 0.63 (0.01) 68 (10%) 4.73 (0.06) 0.53 (0.01) 20 1.3 57.3 (0.14) 4.76 (0.04) 1.19 (0.01) 65 (12%) 3.94 (0.04) 1.17 (0.01) 62 (8%) 4.57 (0.04) 1.14 (0.01) 30 1.0 52.8 (0.12) 4.70 (0.03) 2.01 (0.01) 60 (12%) 3.50 (0.03) 2.01 (0.02) 57 (7%) 4.81 (0.02) 3.21 (0.02) 40 0.8 48.0 (0.11) 4.74 (0.03) 3.16 (0.02) 56 (14%) 3.00 (0.03) 3.22 (0.02) 50 (3%) 4.81 (0.02) 3.20 (0.02) 50 0.7 45.1 (0.10) 4.47 (0.03) 4.47 (0.03) 53 (15%) 2.53 (0.02) 4.90 (0.03) 47 (4%) 4.39 (0.02) 4.39 (0.02) 60 0.8 48.2 (0.11) 3.19 (0.02) 4.79 (0.03) 56 (14%) 2.02 (0.02) 4.94 (0.03) 50 (3%) 3.17 (0.02) 4.76 (0.02) 70 1.0 52.8 (0.12) 2.03 (0.01) 4.74 (0.03) 60 (12%) 1.52 (0.01) 4.74 (0.04) 57 (7%) 2.00 (0.01) 4.59 (0.03) 80 1.3 57.0 (0.13) 1.20 (0.01) 4.78 (0.04) 64 (11%) 1.00 (0.01) 5.00 (0.05) 63 (10%) 1.09 (0.02) 4.38 (0.04) 90 1.9 61.8 (0.16) 0.51 (0.01) 4.63 (0.06) 72 (14%) 0.50 (0.01) 4.87 (0.06) 71 (13%) 0.47 (0.01) 4.24 (0.06) 99 3.9 48.7 (0.21) 0.04 (0.00) 4.10 (0.20) 90 (46%) 0.05 (0.00) 5.62 (0.17) 79 (38%) 0.04 (0.00) 4.13 (0.20) Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 34 modification of their critical values to be asymptotically optimal for control of other metrics, including FDR/FNR and pFDR/pFNR, in the setting of a known number of signals, or bounds on the true number of signals. One interpreta- tion of these results is that first order optimality is not fine grain enough to distinguish between FWE, FDR, pFDR, and other metrics satisfying the main theorems, 2.3.1 and 2.4.1, however it remains an open question whether optimal procedures for control of these metrics must look different asymptotically. The boundedness conditions (i)-(ii) in Theorems 2.3.1 and 2.4.1 are required to hold for all values of the nominal error probabilities a and b, but this is just to obtain admissibility of the gap and gap-intersection rules (Part 1 of Theo- rems 2.3.1 and 2.4.1, respectively) for alla and b, and is not required for asymp- totic optimality (Part 2 of the theorems) which only need to consider a,b near 0. Thus, the optimality results can be further extended to multiple testing error metrics that only satisfy the boundedness conditions (i)-(ii) for smalla,b. Exclusion of the`= 0, u= J cases for pFDR/pFNR control In Section 2.4.1 we have ruled out the cases of the number of signals equal to 0 or J for pFDR/pFNR control based on the reasoning that either of these metrics are not defined for procedures or scenarios in which P(R = 0) = 1 or P(R = J) = 1, which are plausible when there are no, or all, streams contain signals. Focusing on the case of no signals (with similar remarks applying to the other case), the event R= 0 of no rejections has previously been recognized as a difficulty with utilizing pFDR in the fixed-sample analyses, and so is not unique to the sequential sampling considered here. Even in Storey’s (2002) original pro- posal for estimating pFDR, R must be replaced by R_ 1 in the denominator of his estimator to avoid singularity on the event R = 0, and the probability of this event must be bounded below, relying on independence and exact uniform Chapter2. ProcedurestocontrolFDR,pFDR,andrelatedmetrics 35 distribution of the associated p-values. Black (2004, Section 4.3) points out that this approach introduces a bias into the estimate of pFDR and argues that an improved estimator should indeed be left undefined on R = 0. These argu- ments further support our choice to leave out the cases ` = 0 and u = J in Corollary 2.4.2. The other metrics mentioned in Section 2.4.2 to which the general theorem applies, do not have this problem because of elementary bounds like (2.16) (re- placing m by u) and (2.17). 36 Chapter 3 Procedures to control a generalized error metric 3.1 Introduction and Notation In Chapter 2 we discussed procedures to control FDR, pFDR and their related type II analogs. In this chapter, we will control the generalized mis-classification rate proposed by Song and Fellouris (2017), which is the probability of at least k null hypotheses being misclassified below a user specified level a for given 1 k< J anda2(0, 1). That is, we shall consider procedures(T, D) with P A (jD Aj k) a (3.1) where A is the set of signals, denotes the symmetric difference of two sets D A := (Dn A)[(An D), and a2 (0, 1). When k = 1, this recovers the clas- sical mis-classification rate, i.e., the probability of not correctly identifying the true subset of signals, which has been considered by Malloy (2014). The gener- alized error metric unifies type 1 and 2 errors, so the user doesn’t have to specify two criteria (e.g., bounds on both the type 1 and 2 error metrics), which can be difficult for users without expertise in statistics. The paper of Song and Fellouris Chapter3. Procedurestocontrolageneralizederrormetric 37 (2017) takes a completely frequentist approach, without any sort of prior infor- mation the number or identity of the signals (i.e., false nulls). In this chapter, we describe the procedures to control generalized mis-classification error with prior information of the number of signals and we show that utilizing this type of prior information is valuable in terms of efficiency. The case of the number of signals known exactly is addressed in Section 3.2. The case of bounds on the number of signals is addressed in Section 3.3. In Section 3.4 a comparison of the performance of the procedures with and without prior information is stated. Be- yond what is introduced here, all the notation and assumptions in this chapter is the same as in Section 1.3. 3.2 Number of signals known exactly In this section, like in Section 2.3, we consider the setup in which the number of signals m is known, for some 1 m J 1, with the cases m = 0 and m= J being trivial since we can accept all hypotheses when m= 0 and reject all hypotheses when m = J. Thus the relevant signal set isP m =fA [J] :jAj = mg and the class of admissible tests takes the form D a (P m )=f(T, D) : P A (jD Aj k) a f or every A2P m g. In this context, we propose the following leap rule, which will turn out to be asymptotically optimal for generalized error with the number of signals fixed to be m. The leap rule is defined in terms of two other auxiliary procedures, which we define in (3.2) and (3.4) before defining the leap rule in (3.6). Given 0 l< k, Chapter3. Procedurestocontrolageneralizederrormetric 38 consider the procedure( ˇ T l , ˇ D l ) that stops at time ˇ T l := inf ( n 1 : j=ml å j=mlr l +1 l (j) (n) j=m+r l å j=m+1 l (j) (n) c and r l = k l 2 ) , (3.2) and rejects the most significant m l null hypotheses, i.e., ˇ D l := i 1 ( ˇ T l ), ..., i ml ( ˇ T l ) . (3.3) For 0 < l < k, consider a similar procedure ( ˆ T l , ˆ D l ) that rejects the m+ l most significant null hypothesis at time ˆ T l := inf ( n 1 : j=m å j=mr l +1 l (j) (n) j=m+l+r l å j=m+l+1 l (j) (n) c and r l = k l 2 ) , (3.4) with ˆ D l := i 1 ( ˆ T l ), ..., i m+l ( ˆ T l ) . (3.5) The leap rule(T L , D L ) stops as soon as any of these procedures stops, for any l in the ranges above. That is, T L (c)= min min 0l<k ˇ T l , min 0<l<k ˆ T l , (3.6) D L (c)= [ 0l<k ˇ D l [ [ 0<l<k ˆ D l . (3.7) We will include the value of the threshold c as an argument in the procedure(T L (c), D L (c)) when you wish to emphasize the value of the threshold. Next we show how to choose the gap c such that the generalized mis-classification error is controlled for a given value ofa. Chapter3. Procedurestocontrolageneralizederrormetric 39 3.2.1 Error control for the leap rule The next lemma shows how to find the threshold c to guarantee the desired error control for the leap rule. Lemma 3.2.1. Assume the streams (1.3) are independent. For given a2 (0, 1) and m2[J 1], if c= j logaj+ log l=k1 å l=0 m l r l J m r l + l=k1 å l=1 m r l J m l r l ! , (3.8) then (T L (c), D L (c))2D a (P m ). Proof. Fix l and k such that 0 l < k. If the procedure rejects the m l hy- potheses with the largest log-likelihood ratios with r l false positives, there are m l r l true positives and hence m(m l r l ) = l+ r l false negatives. If j ˇ D l Aj k, then r l + l+ r l k, thus r l (k l)/2. Fix A2P m , then there exists a set of false negatives B 1 A c (the latter denoting the complement of A) withjB 1 j= l+ r l and a set of false positives B 2 A withjB 2 j= r l . Then besides the l hypotheses get accepted with largest log-likelihood ratios, there exists set of false negatives M B 1 withjMj= r l . Then fj ˇ D l Aj kg [ B 2 2 A and M2 A c andjB 2 j= r l andjMj= r l with r l = k l 2 , (3.9) where the union is over all possible selection of set M and B 2 in A c and A. With a change of measure P A ! P C and C = (An B 2 )[ M, from (1.9) and Wald’s Chapter3. Procedurestocontrolageneralizederrormetric 40 likelihood ratio identity it follows that P A (fj ˇ D l Aj kg) m l r l J m r l E C [expf å j2B 2 l j å j2M l j g] = m l r l J m r l e c . (3.10) Since for any pair of log likelihood ratios whose indices are in B 2 and M, respec- tively, the gap between them is larger than c. Similarly P A (fj ˆ D l Aj kg) l=k1 å l=1 m r l J m l r l e c . (3.11) Combining (3.10) and (3.11) we have P A (fjD L Aj kg) l=k1 å l=0 P A (j ˇ D l Aj k)+ l=k1 å l=1 P A (j ˆ D l Aj k) l=k1 å l=0 m l r l J m r l e c + l=k1 å l=1 m r l J m l r l e c l=k1 å l=0 m l r l J m r l + l=k1 å l=1 m r l J m l r l ! e c a. (3.12) 3.2.2 Asymptotic optimality of the leap rule In this section show that the leap rule is asymptotically optimality in the sense of asymptotically minimizing the expected sample size as a goes to 0 among all procedures whose generalized error metric is no larger than a. For this we first need to obtain an asymptotic lower bound on the optimal expected sample size for procedures in the classD a (P m ). Let A be the set of signals and Chapter3. Procedurestocontrolageneralizederrormetric 41 I (1) 0 I (2) 0 ... I (Jm) 0 and I (1) 1 I (2) 1 ... I (m) 1 are the ordered information num- bers defined in (1.3). Define some constants based on the information numbers of the data streams under P A : ˇ h l (A, k)= j=r l å j=1 (I (j) 0 + I (j+l) 1 ), ˆ h l (A, k)= j=r l å j=1 (I (j+l) 0 + I (j) 1 ) and r l = k l 2 . (3.13) h L (A, k)= max max 0l<k ˇ h l (A, k), max 0<l<k ˆ h l (A, k) . (3.14) In order to prove the lower bound on the expected sample size of the proce- dures in the admissible classD a (P m ), we use an argument similar to that of Song and Fellouris (2017), as follows. For each “correct” subset B withjB Aj < k, we can choose B where(i) B is “incorrect” under P B , i.e.,jB B j k, and(ii) B is close to A, in the sense that I A,B h L (A, k), where I A,B is defined sim- ilar to I A,C in (1.10). The existence of such a B is guaranteed by the following lemma. Lemma 3.2.2. Fix m2[J 1]. Let A, B[J] be such thatjAj= m andjB Aj< k. Then there exists B [J] withjB j= m such that: (i)jB B j k, (ii) I A,B h L (A, k). Proof. Letting a =jA/Bj and b =jB/Aj, let C A\ B be the indices of the smallest r = l kab 2 m information numbers in A\ B. Similarly, let D(A[ B) c be the indices of the smallest r information numbers in(A[ B) c . Then set B = (A/B)[(A\ B/C)[ D. We see that B B = (A/B)[(B/A)[ C[ D, thus Chapter3. Procedurestocontrolageneralizederrormetric 42 jB B j= a+ b+ 2r k and A B = C[ D. Then I A,B = I C + I D j=r å j=1 I (a+j) 1 + I (b+j) 0 . (3.15) If a< b, then with l = b a, j=r å j=1 (I (a+j) 1 + I (b+j) 0 )= j=d k+ab 2 e å j=a+1 I 1 + j=d k+ba 2 e å j=b+1 I 0 j=d k+ab 2 e å j=1 I 1 + j=d k+ba 2 e å j=ba+1 I 0 = ˆ h l (A, k). Similarly, if a b,å j=r j=1 (I (a+j) 1 + I (b+j) 0 ) ˇ h l (A, k) with l = a b, thus I A,B h L (A, k). With the Lemma 3.2.2 above, we are now ready to prove Theorem 3.2.1 which establishes lower bound on the expected sample size of the procedures in the admissible classD a (P m ). Theorem 3.2.1. Suppose the data streams (1.3) are independent and the assumption of SLLN (1.6) holds, Then for any A[J] withjAj= m, we have, asa! 0, inf (T,D)2D a (P m ) E A [T] j log(a)j h L (A, k) (1+ o(1)). (3.16) Proof. The proof is similar to Song and Fellouris (2016, Theorem 3.2), but here we restrict our consideration to signal sets A withjAj = m. Their proof estab- lishes an asymptotic lower bound for arbitrary A, and we modify and improve the 1st order constant by restricting tojAj = m, for which we choose B with jB j = m, and let h L (A, k) in place of theirD(A, k). All the other steps in the proof remain the same, so we omit the details here. Chapter3. Procedurestocontrolageneralizederrormetric 43 The following lemma establishes an asymptotic upper bound for the ex- pected sample size of the leap rule. Lemma 3.2.3. Assume SLLN (1.6) holds, for any A [J] withjAj = m, as c! ¥ and E A [ ˇ T l ] c(1+ o(1)) ˇ h l (A, k) for every 0 l< k, and E A [ ˆ T l ] c(1+ o(1)) ˆ h l (A, k) for every 0< l< k. Proof. Fix a signal set A [J] satisfyingjAj = m and define the following classes of subsets: M 1 =fB A :jBj= r l g , M 0 =fB A c :jBj= r l , I i 0 I l+1 0 ,8i2 Bg. We define ˆ T 0 l := inf ( n 1 : min B 1 2M 1 ,B 0 2M 0 å j2B 1 l j (n) å j2B 0 l j (n) c ) = inf n 1 : j=m å j=mr l +1 l (j) (n) j=m+l+r l å j=m+l+1 l (j) (n) c, i 1 (n)2 A, ..., i m (n)2 A . (3.17) Then ˆ T l ˆ T 0 l a.s. because the additional requirement of i 1 (n)2 A, ..., i m (n)2 A for ˆ T 0 l . Thus, we have E A [ ˆ T l ] E A [ ˆ T 0 l ] c(1+ o(1)) ˆ h l . (3.18) Chapter3. Procedurestocontrolageneralizederrormetric 44 Similarly, we have E A [ ˇ T l ] c(1+ o(1)) ˇ h l . (3.19) Thus E A [T L ] min min 0l<k c(1+ o(1)) ˇ h l , min 0<l<k c(1+ o(1)) ˆ h l = c(1+ o(1)) h L (A, k) . (3.20) Based on Theorem 3.2.1 and Lemma 3.2.3, we are now ready to establish the asymptotic optimality of the leap rule. Theorem 3.2.2. Assume the data streams (1.3) are independent and the assumption of SLLN (1.6) holds. Then for any A withjAj= m and A[J], we have that asa! 0, E A [T L ] j log(a)j h L (A, k) (1+ o(1)) inf (T,D)2D a (P m ) E A [T]. Proof. Lemma 3.2.1 shows that c=j logaj+ log l=k1 å l=0 m l r l J m r l + l=k1 å l=1 m r l J m l r l ! j log(a)j. (3.21) Then from Lemma 3.2.3, E A [T L ] c(1+ o(1)) h L (A, k) j log(a)j(1+ o(1)) h L (A, k) . (3.22) And from Theorem 3.2.1 inf (T,D)2D a (P m ) E A [T] j log(a)j h L (A, k) (1+ o(1)). (3.23) Chapter3. Procedurestocontrolageneralizederrormetric 45 The lower bound is the same order as the upper bound, which proves the leap rule attains the asymptotic lower bound. 3.3 Bounds on the number of signals In this section, we will consider prior information in the form of bounds on the number of signals. That is, we assume that we know values 0`< u J such that the true number of signals is at least` and at most u. The case` = u has already been discussed in Section 3.2 with the fixed number of signals. This setup includes the case ` = 0 and u = J, i.e., no prior information about the number of signals. Thus the relevant signal set isP `,u =fA[J] :`jAj ug and the relevant class of admissible tests takes the form: D a (P `,u )=f(T, D) : P A (jD Aj k) a for every A2P `,u g. In this class, (T, D) is any measurable procedure as defined in Section 1.3. Fix `, u as above for the remainder of this chapter. We will define a new procedure (T LS , D LS ) in (3.30) which we call the leap sum rule that is asymptotically optimal with prior information in the form of bounds on the number of signals. The leap sum rule is defined in terms of the following sequence of auxiliary procedures. For given m with` m u, 0 t< k and 0 s m t let ˇ T t,s = inf 8 < : n 1 : j=mt å j=mts+1 l (j) (n) j=m+ ˇ f(t,s) å j=m+1 l (j) (n) c 9 = ; , where the l (j) (n) are log-likelihood ratios’ order statistics as defined in (1.7). In this expression ˇ f(t, s)= min i fi :` m s+ i u and s+ i k t and 0 i J mg, Chapter3. Procedurestocontrolageneralizederrormetric 46 thus ˇ f(t, s) takes the minimal value in the set defined above. For this, and other related stopping times, we refer to t as the leap size because we take a leap of size t when substract the log-likelihood ratios’ order statistics. Now define a procedure with fixed leap size t: ˇ T t = max 0smt f ˇ T t,s g, (3.24) ˇ D t =fi 1 ( ˇ T t ), ..., i mt ( ˇ T t )g. (3.25) This procedure rejects m t hypotheses with the largest log-likelihood ratios when the gap is large than c for every s with 0 s m t. Similarly, for 0< t< k and 0 s m define ˆ T t,s = inf 8 < : n 1 : j=m å j=ms+1 l (j) (n) j=m+t+ ˆ f(t,s)+1 å j=m+t+1 l (j) (n) c 9 = ; , In this expression ˆ f(t, s)= min i fi :` m s+ i u and s+ i k t and 0 i J m tg, thus ˇ f(t, s) takes the minimal value in the set defined above. Now define a procedure with fixed leap size t: ˆ T t = max 0sm ˆ T t,s , (3.26) ˆ D t = i 1 ( ˆ T t ), ..., i m+t ( ˆ T t ) . (3.27) This procedure rejects m+ t hypotheses with the largest log-likelihood ratios when the gap is large than c for every s with 0 s m. Chapter3. Procedurestocontrolageneralizederrormetric 47 Next we define the stopping time for a target number m of signals as T m = min min 0t<k ˇ T t , min 0<t<k ˆ T t , (3.28) D m = [ 0t<k ˇ D t [ [ 0<t<k ˆ D t . (3.29) This procedure combines ˇ T t and ˆ T t and stops as soon as any of these procedures does so, and use the corresponding decision rule upon stopping. We now finally define the leap sum rule as stopping the first time any corre- sponding stopping time T m stops, for any target number of signals` m u. That is, for` m u, the stopping time for the leap sum rule will be: T LS = min `mu fT m g, (3.30) D LS = [ `mu D m . (3.31) 3.3.1 Error control for the leap sum rule We proceed to show how to find the threshold c to guarantee the desired error control for the leap sum rule. Lemma 3.3.1. Assume the streams (1.3) are independent, given 0 < a < 1 and 0 < `< u< J,(T LS , D LS )2D a (P `,u ) if c= j log(a)j+ log m=u å m=` t=k1 å t=0 s=mt å s=0 m t s J m ˇ f(t, s) + t=k1 å t=1 s=m å s=0 m s J m t ˆ f(t, s) . (3.32) Proof. Fix t with 0 t < k and m with` m u. If the procedure rejects m t hypotheses with the largest log-likelihood ratios, with s false positives, Chapter3. Procedurestocontrolageneralizederrormetric 48 the minimal number ˇ g(t, s) of false negatives is: ˇ g(t, s)= min i f` i+ m t s u and 0 i J m+ tg. Since the number of hypothesis that get rejected is m t and the number of false positives is s, then the number of true positives is m t s. If the number of false negatives is i, we have ` i+ m t s u and also number of false negative should be smaller than the number of hypothesis get accepted, thus 0 i J m+ t. Then the minimal number of false negatives besides the first t is ˇ h(t, s)= max( ˇ g(s) t, 0) which is ˇ h(t, s)= min i f` i+ m s u and 0 i J mg. IfjD Aj is larger than or equal to k, with s false positives, the minimal number of false negatives besides the first t is ˇ f(t, s)= min i f` i+ m s u and i+ s k t and 0 i J mg. Fix A [J] with |A| = m, then there exists a set of false negatives B 1 A c (the latter denoting the complement of A) withjB 1 j = t+ ˇ f(t, s) and a set of false positives B 2 A withjB 2 j = s. Then besides the first t hypotheses that get accepted with largest log-likelihood ratios, there exists M B 1 withjMj = ˇ f(t, s). Then fj ˇ D t Aj kg [ n B 2 2 A and M2 A c andjB 2 j= s andjMj= ˇ f(t, s) o . The union is over all possible selection of set M and B 2 . With a change of mea- sure P A ! P C and C = (An B 2 )[ M, from (1.9) and Wald’s likelihood ratio Chapter3. Procedurestocontrolageneralizederrormetric 49 identity it follows that P A (fj ˇ D t Aj kg) m t s J m ˇ f(t, s) E C [expf å j2B 2 l j å j2M l j g] m t s J m ˇ f(t, s) e c . (3.33) Since for any pair of log likelihood ratios whose indices are in B 2 and M, respec- tively, the gap between them is larger than c. The inequality holds for any s with 0 s m t, then we have P A (fj ˇ D t Aj kg) s=mt å s=0 m t s J m ˇ f(t, s) e c . (3.34) Similarly P A (fj ˆ D t Aj kg) s=m å s=0 m s J m t ˆ f(t, s) e c . (3.35) Combining (3.34) and (3.35) we have P A (fjD m Aj kg) t=k1 å t=0 s=mt å s=0 m t s J m ˇ f(t, s) e c + t=k1 å t=1 s=m å s=0 m s J m t ˆ f(t, s) e c . (3.36) Finally, P A (fjD LS Aj kg) m=u å m=` P A (fjD m Aj kg) m=u å m=` t=k1 å t=0 s=mt å s=0 m t s J m ˇ f(t, s) + t=k1 å t=1 s=m å s=0 m s J m t ˆ f(t, s) ! e c . (3.37) Chapter3. Procedurestocontrolageneralizederrormetric 50 3.3.2 Asymptotic optimality for the leap sum rule In this section show that the leap rule is asymptotically optimality in the sense of asymptotically minimizing the expected sample size as a goes to 0 among all procedures whose generalized error metric is no larger than a. For this we first need to obtain an asymptotic lower bound on the optimal expected sample size for procedures in the classD a (P `,u ). Let A be the the set of signals and we define some constants based on the information numbers of the data streams under P A : ˇ h t,s (A, k)= j=s å j=1 I (t+j) 1 + j= ˇ f(t,s) å j=1 I (j) 0 . (3.38) And we denote ˇ h t (A, k)= min 0smt ˇ h t,s (A, k). (3.39) Similarly, define ˆ h t,s (A, k)= j=s å j=1 I (j) 1 + j= ˆ f(t,s) å j=1 I (t+j) 0 . (3.40) And ˆ h t (A, k) := min 0sm ˆ h t,s (A, k). (3.41) Finally, let h m (A, k) := max max 0t<k ˇ h t (A, k), max 0<t<k ˆ h t (A, k) . (3.42) In order to prove the lower bound on the expected sample size of the pro- cedures in the admissible classD a (P `,u ), we use an argument similar to that of Song and Fellouris (2017), as follows. For each “correct” subset B withjB Aj< k, we can choose B where(i) B is “incorrect” under P B , i.e.,jB B j k, and (ii) B is close to A, in the sense that I A,B h m (A, k), where I A,B is defined Chapter3. Procedurestocontrolageneralizederrormetric 51 similar to I A,C in (1.10). The existence of such a B is guaranteed by the following lemma. Lemma 3.3.2. Let A, B [J] andjAj = m with m2 [J 1] andjB Aj< k. Then there exists B [J] with`jB j u such that: (i)jB B j k, (ii) I A,B h m (A, k). Proof. LetjA/Bj = a andjB/Aj = b. If a < b, let C A\ B be the indices of the smallest s information numbers in A\ B. Similarly, let D (A[ B) c be the indices of the smallest ˆ f(a+ b, s) information numbers in(A[ B) c . Then set B 0 = (An B)[((A\ B)n C)[ D. We can see that B B 0 =(An B)[(Bn A)[ C[ D, thusjB B 0 j= a+ b+ s+ ˆ f(a+ b, s) k and A B 0 = C[ D then I A,B 0 = I C + I D j=s å j=1 I (a+j) 1 + j= ˆ f(a+b,s) å j=1 I (b+j) 0 ˆ h ba,s+a (A, k). (3.43) Since ˆ h ba,s+a (A, k)= j=s+a å j=1 I (j) 1 + j=ba+ ˆ f(ba,s+a) å j=ba+1 I (j) 0 . (3.44) and ˆ f(b a, s+ a)= ˆ f(a+ b, s)+ a, then ˆ h ba,s+a (A, k)= j=s+a å j=1 I (j) 1 + j=b+ ˆ f(a+b,s) å j=ba+1 I (j) 0 . (3.45) For 0 s m, let B = B 0 with the smallest ˆ h ba,s (A, k). Then I A,B min 0sm ˆ h ba,s (A, k)= ˆ h ba (A, k). (3.46) Chapter3. Procedurestocontrolageneralizederrormetric 52 Then similarly, if a b, j=s å j=1 I (a+j) 1 + j= ˇ f(a+b,s) å j=1 I (b+j) 0 ˇ h ab,s+b (A, k). (3.47) Thus we have I A,B min 0smt ˇ h ab,s (A, k)= ˇ h ab (A, k). (3.48) Finally for any pair of values for a and b, we have I A,B maxf max 0t<k ˇ h t (A, k), max 0<t<k ˆ h t (A, k)g h m (A, k). (3.49) With Lemma 3.3.2 established, we are now ready to prove Theorem 3.3.1 which establishes lower bound on the expected sample size of the procedures in the admissible classD a (P `,u ). Theorem 3.3.1. Suppose the data streams (1.3) are independent and the assumption of SLLN (1.6) holds, then for any A[J] withjAj= m, we have asa! 0 inf (T,D)2D a (P `,u ) E A [T] c h m (A, k) (1+ o(1)). Proof. The proof is a modification of the proof of Song and Fellouris (2016, Theorem 3.2) by restricting the set with signals A to bejAj= m. In their proof A is arbitrary, but here withjAj= m, when we choose B , we require`jB j u and the resulting constant in the proof of Song-FellourisD(A, k) become the new constant h m (A, k). The other steps in the proof are largely the same as in Song-Fellouris and so we omit them here. The following lemma establishes an asymptotic upper bound for the ex- pected sample size of the leap sum rule. Chapter3. Procedurestocontrolageneralizederrormetric 53 Lemma 3.3.3. Assume SLLN (1.6) holds, for any A [J] withjAj = m, as c! ¥ and E A [ ˇ T t ] c(1+ o(1)) ˇ h t (A, k) for every 0 t< k, and E A [ ˆ T t ] c(1+ o(1)) ˆ h t (A, k) for every 0< t< k. Proof. Fix A, we introduce the following classes of subsets: M 1 =fB A :jBj= sg, M 0 =fB A C :jBj= ˆ f(t, s), I i 0 I t+1 0 ,8i2 Bg. We define ˆ T 0 t,s and similar to the proof of Lemma 3.2.3, we have ˆ T t,s ˆ T 0 t,s a.s. ˆ T 0 t,s = inf ( n 1 : min B 1 2M 1 ,B 0 2M 0 å j2B 1 l (j) (n) å j2B 0 l (j) (n) c ) = inf n 1 : j=m å j=ms+1 l (j) (n) j=m+t+ ˆ f(t,s) å j=m+t+1 l (j) (n) c, i 1 (n)2 A, ..., i m (n)2 A . (3.50) Thus, we have E A [ ˆ T t,s ] E A [ ˆ T 0 t,s ] c(1+ o(1)) å j=s j=1 I (j) 1 +å j= ˆ f(t,s) j=1 I (j+t) 0 = c(1+ o(1)) ˆ h t,s (A, k) . (3.51) Then, we have E A [ ˆ T t ] max 0sm E A [ ˆ T t,s ]= c(1+ o(1)) ˆ h t (A, k) . (3.52) Chapter3. Procedurestocontrolageneralizederrormetric 54 Similarly, E A [ ˇ T t ] max 0smt E A [ ˇ T t,s ]= c(1+ o(1)) ˇ h t (A, k) . (3.53) Thus for any` m u, we have E A [T LS ]= min `mu E A [T m ] E A [T m ]. (3.54) And we have E A [T m ]= E A [min min 0t<k ˇ T t , min 0<t<k ˆ T t ] c(1+ o(1)) maxfmax 0t<k ˇ h t (A, k), max 0<t<k ˆ h t (A, k)g = c(1+ o(1)) h m (A, k) . (3.55) Using Theorem 3.3.1 and Lemma 3.3.3, we are now ready to establish the asymptotic optimality of the leap sum rule. Theorem 3.3.2. Assume the data streams1.3 are independent and the assumption of SLLN (1.6) holds. Then for any A with`jAj u and A [J], we have that as a! 0, E A [T LS ] j log(a)j h m (A, k) (1+ o(1)) inf (T,D)2D a (P `,u ) E A [T] ifjAj= m. Proof. Lemma 3.3.1 shows that c= j log(a)j+ log m=u å m=` t=k1 å t=0 s=mt å s=0 m t s J m ˇ f(t, s) + t=k1 å t=1 s=m å s=0 m s J m t ˆ f(t, s) j log(a)j. (3.56) Chapter3. Procedurestocontrolageneralizederrormetric 55 Then from Lemma 3.3.3, E A [T LS ] c(1+ o(1)) h m (A, k) j log(a)j(1+ o(1)) h m (A, k) . (3.57) And from Theorem 3.3.1 inf (T,D)2D a (P `,u ) E A [T] j log(a)j h m (A, k) (1+ o(1)). (3.58) The lower bound is same order as the upper bound, which proves the leap rule attains the asymptotic lower bound. 3.4 Effects of prior information In this section we compare the effects of the varying degrees of prior infor- mation in the procedures proposed above for controlling the generalized error metric. This is done first through Theorem 3.4.1, which establishes that, not surprisingly, the limiting expected sample sizes decrease as more precise prior information is considered. Next, for a more detailed, finite-sample comparison, we present a simulation study of the procedures in Table 3.1. Theorem 3.4.1. Assume the data streams 1.3 are independent and the assumption (1.6) of SLLN holds. Fix arbitrary A [J] and let m =jAj and 0 ` m u J. For a2 (0, 1), we consider the following procedures: Let T L denote the leap rule with parameters a and m; let T LS denote the leap sum rule with parameters a,`, and u; and let T S denote the sum rule with parameter a. Then as a! 0, the first order terms in the asymptotic expansions of the expected stopping time E A [T] decrease with more information about the number of signals. That is, E ¥ A [T L ] E ¥ A [T LS ] E ¥ A [T S ]. Chapter3. Procedurestocontrolageneralizederrormetric 56 where E ¥ A (T) :=j logaj lim a!0 E A (T) j logaj is the first order term in the asymptotic expansion of the expected sample size of the procedure T = T(a). Proof. In the leap sum rule, ifjAj= m, h m (A, k)= max max 0t<k ˇ h t (A, k), max 0<t<k ˆ h t (A, k) . And in the leap rule, h L (A, k)= max max 0l<k ˇ h l (A, k), max 0<l<k ˆ h l (A, k) . For each fixed t= l, we have in leap sum rule ˇ h t (A, k)= min 0smt j=s å j=1 I (t+j) 1 + j= ˇ f(t,s) å j=1 I (j) 0 . Finally, in the leap rule ˇ h t (A, k)= j=r å j=1 (I (j) 0 + I (j+t) 1 ), r = k t 2 . If s= r then ˇ f(t, s)= k t r r, so ˇ h t (A, k) in the leap sum rule is less than or equal to ˇ h t (A, k) in leap rule. Thenh L (A, k) h m (A, k), thus E A [T L ] E A [T LS ]. For the second inequality, let h S (A, k) = å j=k j=1 ˜ I (j) denote the information number in the sum rule. In the leap sum rule, h m (A, k)= max max 0t<k ˇ h t (A, k), max 0<t<k ˆ h t (A, k) ˇ h 0 (A, k). Chapter3. Procedurestocontrolageneralizederrormetric 57 We have ˇ h 0 (A, k)= min 0sm j=s å j=1 I (j) 1 + j= ˇ f(0,s) å j=1 I (j) 0 . Since s+ ˇ f(0, s) k, we have ˇ h 0 (A, k) h S (A, k). Thus h m (A, k) h S (A, k). Then we have E A [T LS ] E A [T S ]. Next we present a simulation study of the finite-sample performance of the procedures in the theorem. We use the same normal mean setting we used in Section 2.5. We consider J = 10 independent streams X j 1 , X j 2 , . . ., j2 [J], of i.i.d. N(m, 1) data with hypotheses H j 0 : m = 0 vs. H j 1 : m = 1/2, j2 [J]. We set the number of signals in these data streams to bejAj = 5, and the identities of the signal streams is chosen arbitrarily since the procedures’ performance is invariant with respect to this. In this case the information numbers are I j 0 = I j 1 = 1/8. Furthermore we set a = 0.05, and for each procedure a value of c was determined by Monte Carlo simulation such that the achieved generalized error are close to, but no larger than, this value. The procedures in the table are as follows. We take the number of signals to be m= 5 and consider bounds(`, u)= (0, 10),(1, 9), . . . ,(4, 6),(5, 5) on the number of signals. For the cases with 0 < ` < u < 10, the procedure T is the leap sum rule with parameters` and u. For the case with` = u = m = 5, the procedure is the leap rule with parameters m. The case with` = 0 and u = 10 is the sum rule. For each of these procedures, the expected stopping time ET and achieved generalized error, estimated from 10,000 Monte Carlo replications, are given in the table; standard errors are given in parentheses. The first order terms in the asymptotic expansions (computed exactly) as in Equations (3.14) and (3.42) are also computed accordingly and given in the table. From the table, we can see the expected sample size E[T] decrease as the bounds(`, u) get tighter around m and there is a substantial decrease in sample Chapter3. Procedurestocontrolageneralizederrormetric 58 ` u c E[T] Computed first order term Generalized error 0 10 3.5 76.7 (7.07) 24 0.043 (0.002) 1 9 3.3 74.09 (6.62) 24 0.049 (0.002) 2 8 3.3 74.24 (6.64) 24 0.050 (0.002) 3 7 3.3 73.88 (7.12) 24 0.050 (0.002) 4 6 3.5 73.76 (6.89) 24 0.049 (0.002) 5 5 3.6 42.8 (2.77) 12 0.050 (0.002) TABLE 3.1: Comparison of expected stopping times for procedures with varying degrees of prior information, as described in Sec- tion 3.4 with J = 10 data streams. size for ` = u = m = 5 case. This behavior of the expected sample size is mirrored by the first order asymptotic terms given in the table. Thus, there is a relatively large “cost” (in terms of average sample size) to the leap sum rule when the true number of signals is strictly between the bounds` <jAj < u, because this procedure must always consider the possibility thatjAj = ` or u, versus the leap rule with m = jAj. But perhaps surprisingly, this cost is relatively insensitive to how tight these bounds`, u are around m, until`= u= m, with even the case` = m 1, u = m+ 1 being not much different than cases with substantially smaller` and larger u. 59 Chapter 4 Procedures to control k-family-wise error rates 4.1 Introduction and Notation In Chapters 2 and 3 we have discussed procedures that control FDR, pFDR, and a generalized error metric, respectively. The generalized error metric con- sidered in Chapter 3 combines the type 1 and 2 errors (e.g., false positives and false negatives) into a single metric. This may be appealing in applications when some notion of overall error control is desired, but separate type 1 and 2 levels are not well motivated. On the other hand, in some applications we may have very different desired levels to these two kinds of errors. Thus, in this chapter, we consider procedures that control the probabilities of at least k 1 false posi- tives and at least k 2 false negatives below user-specified levels a and b with 1 k 1 < J, 1 k 2 < J and a2 (0, 1), b2 (0, 1). That is, we shall consider procedures(T, D) with P A (jDn Aj k 1 ) a and P A (jAn Dj k 2 ) b, (4.1) Chapter4. Procedurestocontrol k-family-wiseerrorrates 60 where A is the set of signals and Dn A denotes the difference between two sets. When k 1 = 1 and k 2 = 1, this recovers the classical family-wise error rates, which has been considered in the sequential setup by Song and Fellouris (2016), of which the work in this chapter is an extension. The paper of Song and Fel- louris (2017) takes a completely frequentist approach, without any sort of prior information the number or identity of the signals (i.e., false nulls). In this chap- ter, we describe procedures that control k-family-wise error rates with prior in- formation of the number of signals and we show that utilizing this type of prior information is valuable in terms of efficiency. The case of the number of signals known exactly is addressed in Section 4.2. The case of bounds on the number of signals is addressed in Section 4.3. In Section 4.4 a comparison of the perfor- mance of the procedures with and without prior information is stated. Beyond what is introduced here, all the notation and assumption in this chapter is the same as in Section 1.3. 4.2 Number of signals known exactly In this section, like in Section 2.3, we consider the setup in which the number of signals m is known, for some 1 m J 1, with the cases m = 0 and m= J being trivial since we can accept all hypotheses when m= 0 and reject all hypotheses when m = J. Thus the relevant signal set isP m =fA [J] :jAj = mg and the class of admissible tests takes the form D a,b (P m )=f(T, D) : P A (jDn Aj k 1 ) a and P A (jAn Dj k 2 ) b and every A2P m g. In this context, we propose the following family-wise leap rule, which will turn out to be asymptotically optimal for k-family-wise errors with the number Chapter4. Procedurestocontrol k-family-wiseerrorrates 61 of signals fixed to be m. The family-wise leap rule is defined in terms of the following sequence of auxiliary procedures. Given 0 l < k 2 , consider the procedure( ˇ T l , ˇ D l ) that stops at time ˇ T l = inf n 1 : j=ml å j=mlk 1 +1 l (j) (n) j=m+k 1 å j=m+1 l (j) (n) b and j=ml å j=mk 2 +1 l (j) (n) j=m+k 2 l å j=m+1 l (j) (n) a , (4.2) and rejects the most significant m l null hypotheses, i.e., ˇ D l = i 1 ( ˇ T l ), ..., i ml ( ˇ T l ) . (4.3) For 0< l < k 1 , consider a similar procedure( ˆ T l , ˆ D l ) that rejects the m+ l most significant null hypothesis at time ˆ T l = inf n 1 : j=m å j=mk 1 +l+1 l (j) (n) j=m+k 1 å j=m+l+1 l (j) (n) b and j=m å j=mk 2 +1 l (j) (n) j=m+l+k 2 å j=m+l+1 l (j) (n) a , (4.4) with ˆ D l =fi 1 ( ˆ T l ), ..., i m+l ( ˆ T l )g. (4.5) The family-wise leap rule (T FL , D FL ) stops as soon as any of these procedures stops, for any l in the ranges above. That is, T FL = min min 0l<k 2 ˇ T l , min 0<l<k 1 ˆ T l , (4.6) D FL = [ 0l<k 2 ˇ D l [ [ 0<l<k 1 ˆ D l . (4.7) Chapter4. Procedurestocontrol k-family-wiseerrorrates 62 Next we show how to choose the values of a and b such that the k-family-wise errors are controlled for given values ofa and b. 4.2.1 Error control for the family-wise leap rule The next lemma shows how to find the thresholds a and b to guarantee the desired error control for the family-wise leap rule. Lemma 4.2.1. Assume the streams (1.3) are independent. Givena2(0, 1), b2(0, 1) and m2[J 1], if b=j log(a)j+ log l=k 2 1 å l=0 m l k 1 J m k 1 + l=k 1 1 å l=1 m k 1 l J m l k 1 l ! , (4.8) a=j log(b)j+ log l=k 2 1 å l=0 m l k 2 l J m k 2 l + l=k 1 1 å l=1 m k 2 J m l k 2 ! , (4.9) then (T FL , D FL )2D a,b (P m ). (4.10) Proof. For a fixed l and k 2 such that 0 l< k 2 . If the procedure rejects the first m l hypotheses, for r l false positives, there are m l r l true positives, then there are m(m l r l ) = l+ r l false negatives. Ifj ˇ D l n Aj k 1 , then r l k 1 . Fix A2P m , then there exists a set of false negatives B 1 A c (the latter denoting the complement of A) withjB 1 j = l+ r l and a set of false positives B 2 A with jB 2 j= r l . Then besides the l hypotheses get accepted with largest log-likelihood ratios, there exists set of false negatives M B 1 withjMj= r l . Then fj ˇ D l n Aj k 1 g [ fB 2 2 A and M2 A c andjB 2 j= k 1 andjMj= k 1 g , Chapter4. Procedurestocontrol k-family-wiseerrorrates 63 where the union is over all possible selection of set M and B 2 in A c and A. And similarly fjAn ˇ D l j k 2 g [ fB 2 2 A and M2 A c andjB 2 j= k 2 l andjMj= k 2 lg . With a change of measure P A ! P C with C =(An B 2 )[ M from (1.9) and Wald’s likelihood ratio identity it follows that P A (fj ˇ D l n Aj k 1 g) m l k 1 J m k 1 E C [expf å j2B 2 l j å j2M l j g] m l k 1 J m k 1 e b , (4.11) and P A (fjAn ˇ D l j k 2 g) m l k 2 l J m k 2 l E C [expf å j2B 2 l j å j2M l j g] m l k 2 l J m k 2 l e a . (4.12) Similarly we have P A (fj ˆ D l n Aj k 1 g) ( m k 1 l )( Jml k 1 l )e b and P A (fjAn ˆ D l jg) ( m k 2 )( Jml k 2 )e a . Thus P A (fjD FL n Aj k 1 g) l=k 2 1 å l=0 m l k 1 J m k 1 e b + l=k 1 1 å l=1 m k 1 l J m l k 1 l e b l=k 2 1 å l=0 m l k 1 J m k 1 + l=k 1 1 å l=1 m k 1 l J m l k 1 l ! e b a, (4.13) Chapter4. Procedurestocontrol k-family-wiseerrorrates 64 and P A (fjAn D FL j k 2 g) l=k 2 1 å l=0 m l k 2 l J m k 2 l e a + l=k 1 1 å l=1 m k 2 J m l k 2 e a l=k 2 1 å l=0 m l k 2 l J m k 2 l + l=k 1 1 å l=1 m k 2 J m l k 2 ! e a b. (4.14) 4.2.2 Asymptotic optimality of the family-wise leap rule In this section show that the leap rule is asymptotically optimality in the sense of asymptotically minimizing the expected sample size as a and b go to 0, among all procedures whose k-family-wise error metrics are no larger than a and b, respectively. For this we first need to obtain an asymptotic lower bound on the optimal expected sample size for procedures in the classD a,b (P m ). Let A be the the set of signals and I (1) 0 I (2) 0 ... I (Jm) 0 and I (1) 1 I (2) 1 ... I (m) 1 are the ordered information numbers defined in (1.3). Define the following constants based on the information numbers of the data streams under P A : ˇ h l,1 (A)= j=k 1 å j=1 I (l+j) 1 + j=k 1 å j=1 I (j) 0 . ˆ h l,1 (A)= j=k 1 l å j=1 I (j) 1 + j=k 1 l å j=1 I (l+j) 0 . ˇ h l,2 (A)= j=k 2 l å j=1 I (l+j) 1 + j=k 2 l å j=1 I (j) 0 . ˆ h l,2 (A)= j=k 2 å j=1 I (j) 1 + j=k 2 å j=1 I (l+j) 0 . (4.15) Chapter4. Procedurestocontrol k-family-wiseerrorrates 65 These are sum of information numbers under the scenario that we reject the most significant m l and m+ l null hypotheses. Next define ˆ L l (A)= j log(a)j ˆ h l,1 (A) _ j log(b)j ˆ h l,2 (A) , (4.16) ˇ L l (A)= j log(a)j ˇ h l,1 (A, k) _ j log(b)j ˇ h l,2 (A) , and (4.17) L(A)= min min 0l<k 2 ˇ T l (A), min 0<l<k 1 ˆ T l (A) . (4.18) In order to prove the lower bound of the admissible classD a,b (P m ), we used a similar idea as the proof of Theorem 3.2.1. For each “correct” subset B with jBn Aj< k 1 andjAn Bj< k 2 , we can choose B 1 and B 2 that satisfy the conditions in the lemma below. These sets are worst case alternatives for the true signal set A. Lemma 4.2.2. Fix m2[J 1]. Let A, B[J] be such thatjAj= m andjBn Aj< k 1 , jAn Bj < k 2 . Then there exist B 1 [J] and B 2 [J] withjB 1 j = m andjB 2 j = m such that: (i) IfjBj k 1 andjB C j k 2 , then there exist B 1 , B 2 [J] such that (1)jBn B 1 j= k 1 andjB 2 n Bj= k 2 , (2) j log(a)j I A,B 1 _ log(b)j I A,B 2 L(A). (ii) IfjBj< k 1 , then there exists B 2 [J] such that (1)jB 2 n Bj= k 2 , (2) j log(b)j I A,B 2 L(A). (iii) IfjB C j< k 2 , then there exists B 1 [J] such that (1)jBn B 1 j= k 2 , Chapter4. Procedurestocontrol k-family-wiseerrorrates 66 (2) j log(a)j I A,B 1 L(A). Proof. Let’s start with the first inequality. Letting a =jA/Bj and b =jB/Aj, let C A\ B be the indices of the smallest k 2 a information numbers in A\ B. Similarly, let D (A[ B) c be the indices of the smallest k 2 a information numbers in (A[ B) c . Then set B 2 = (An B)[((A\ B)n C)[ D. We see that B 2 n B = (An B)[ D, thusjB 2 n Bj = a+ k 2 a = k 2 andjB 2 j = m(k 2 a)+ k 2 a= m. And I A,B 2 = I C + I D . Similarly, we can construct B 1 as follows: let C A\ B be the indices of the smallest k 1 b information numbers in A\ B. Similarly, let D (A[ B) c be the indices of the smallest k 1 b information numbers in (A[ B) c . Then set B 1 = (An B)[((A\ B)n C)[ D. We can see that Bn B 1 = (Bn A)[ C, thusjBn B 1 j = b+ k 1 b = k 1 andjB 1 j = m(k 1 b)+ k 1 b = m. And I A,B 1 = I C + I D . If a b with l = a b then ˇ h ab,1 (A)= j=ab+k 1 å j=ab+1 I (j) 1 + j=k 1 å j=1 I (j) 0 j=ab+k 1 å j=a+1 I (j) 1 + j=k 1 å j=b+1 I (j) 0 I A,B 1 . (4.19) Similarly, we can prove that ˇ h ab,2 I A,B 2 . Thus j log(a)j I A,B 1 _ j log(b)j I A,B 2 ˇ L ab (A). (4.20) If a< b, then with l = b a we can argue similarly that j log(a)j I A,B 1 _ j log(b)j I A,B 2 ˆ L ba (A). Thus j log(a)j I A,B 1 _ j log(b)j I A,B 2 L(A). Chapter4. Procedurestocontrol k-family-wiseerrorrates 67 Then condition (i).(2) is satisfied. The proof of(ii) and(iii) are similar. With the Lemma 4.2.2 established above, we are now ready to prove The- orem 4.2.1 which establishes lower bound on the expected sample size of the procedures in the admissible classD a,b (P m ). Theorem 4.2.1. Suppose the data streams (1.3) are independent and the assumption of SLLN (1.6) holds, Then for any A[J] withjAj= m, we have, asa! 0 and b! 0, inf (T,D)2D a,b (P m ) E A [T] L(A)(1+ o(1)). (4.21) Proof. The proof is similar to Song and Fellouris (2016, Theorem 4.3), but here we restrict our consideration to signal sets A withjAj = m. Their proof estab- lishes an asymptotic lower bound for arbitrary A, and we modify and improve the 1st order constant by restricting tojAj = m, for which we choose B 1 and B 2 withjB 1 j =jB 2 j = m, and let L(A) in place of their L(A;a,b). All the other steps in the proof remain the same, so we omit the details here. The following lemma establishes an asymptotic upper bound for the ex- pected sample size of the family-wise leap rule. Lemma 4.2.3. Assume of SLLN (1.6) holds, for any A [J] withjAj = m, as a, b! ¥ and bj log(a)j and aj log(b)j, we have for every 0 l< k, E A [ ˇ T l ] b(1+ o(1)) ˇ h l,1 (A) _ a(1+ o(1)) ˇ h l,2 (A) , (4.22) and for 0< l< k, E A [ ˆ T l ] b(1+ o(1)) ˆ h l,1 (A) _ a(1+ o(1)) ˆ h l,2 (A) . (4.23) Chapter4. Procedurestocontrol k-family-wiseerrorrates 68 Proof. Fix a signal set A [J] satisfyingjAj = m and define the following classes of subsets: M 1 =fB A, C A c :jBj= k 1 l,jCj= k 1 l, I i 0 I l+1 0 ,8i2 Cg, M 2 =fB A, C A c :jBj= k 2 ,jCj= k 2 , I i 0 I l+1 0 ,8i2 Cg. We define an auxiliary stopping time ˆ T 0 l ˆ T 0 l := inf n 1 : min B,C2M 1 å j2B l j (n) å j2C l j (n) b and and min B,C2M 2 å j2B l j (n) å j2C l j (n) a = inf n 1 : j=m å j=mk 1 +l+1 l (j) (n) j=m+k 1 å j=m+l+1 l (j) (n) b and j=m å j=mk 2 +1 l (j) (n) j=m+l+k 2 å j=m+l+1 l (j) (n) a, i 1 (n)2 A, ..., i m (n)2 A . (4.24) Then ˆ T l ˆ T 0 l a.s. because the additional requirement of i 1 (n)2 A, ..., i m (n)2 A for ˆ T 0 l . Thus, we have E A [ ˆ T l ] E A [ ˆ T 0 l ] b(1+ o(1)) å j=k 1 l j=1 (I (j+l) 0 + I (j) 1 ) _ a(1+ o(1)) å j=k 2 j=1 (I (j+l) 0 + I (j) 1 ) = b(1+ o(1)) ˆ h l,1 (A) _ a(1+ o(1)) ˆ h l,2 (A) . (4.25) Similarly E A [ ˇ T l ] b(1+ o(1)) ˇ h l,1 (A) _ a(1+ o(1)) ˇ h l,2 (A) . (4.26) Chapter4. Procedurestocontrol k-family-wiseerrorrates 69 Thus, since bj log(a)j and aj log(b)j we have E A [T FL ]= min min 0l<k 2 ˇ T l , min 0<l<k 1 ˆ T l L(A)(1+ o(1)). (4.27) Based on Theorem 4.2.1 and Lemma 4.2.3, we are now ready to establish the asymptotic optimality of the family-wise leap rule. Theorem 4.2.2. Suppose the data streams are independent and the assumption of SLLN (1.6) holds, Then for any A[J] withjAj= m, we have asa! 0 and b! 0, E A [T FL ] L(A)(1+ o(1)) inf (T,D)24 a,b (P m ) E A [T]. Proof. The a, b in Lemma 4.2.1 satisfy bj log(a)j and aj log(b)j. And from (4.27), E A [T FL ] L(A)(1+ o(1)). (4.28) And from Theorem 4.2.1 inf (T,D)2D a,b (P m ) E A [T] L(A)(1+ o(1)). (4.29) The lower bound is the same order as the upper bound, which proves the family- wise leap rule attains the asymptotic lower bound. 4.3 Bounds on the number of signals In this section, we will consider prior information in the form of bounds on the number of signals. That is, we assume that we know values 0`< u J such that the true number of signals is at least` and at most u. The case` = u has already been discussed in Section 4.2 with the fixed number of signals. This setup includes the case ` = 0 and u = J, i.e., no prior information about the Chapter4. Procedurestocontrol k-family-wiseerrorrates 70 number of signals. Thus the relevant signal set isP `,u =fA[J] :`jAj ug and the relevant class of admissible tests takes the form: D a,b (P `,u )=f(T, D) : P A (jDn Aj k 1 ) a and P A (jAn Dj k 2 ) b for every A2P `,u g. In this class, (T, D) is any measurable procedure as defiend in Section 1.3. Fix `, u as above for the remainder of this chapter. We will define a new procedure (T FLS , D FLS ) in (4.36) which we call the family-wise leap sum rule that is asymp- totically optimal with prior information in the form of bounds on the number of signals. The leap sum rule is defined in terms of the following sequence of aux- iliary procedures which depend on a candidate number m of signals. This value is of course unknown in this setup (other than being within the bounds`, u, and the family-wise leap sum rule(T FLS , D FLS ) is ultimately defined in (4.36)-(4.37) by taking the minimum and union, respectively, over m. For given m with` m u and 0 t< k 2 , define ˇ T t = inf n 1 : j=mt å j=mtˇ s 1 +1 l (j) (n) j=m+ ˇ i 1 å j=m+1 l (j) (n) b and j=mt å j=mtˇ s 2 +1 l (j) (n) j=m+ ˇ i 2 å j=m+1 l (j) (n) a , (4.30) ˇ D t =fi 1 ( ˇ T t ), ..., i mt ( ˇ T t )g. (4.31) In this expression, ˇ s 1 = k 1 , ˇ i 1 = maxf` m+ k 1 , 0g, ˇ s 2 = maxfm+ k 2 t u, 0g, and ˇ i 2 = k 2 t. This procedure rejects m t hypotheses with the largest log- likelihood ratios when the gaps are large than a and b for every t with 0 t< k 2 . Chapter4. Procedurestocontrol k-family-wiseerrorrates 71 Similarly, for 0< t< k 1 define ˆ T t := inf n 1 : j=m å j=mˆ s 1 +1 l (j) (n) j=m+t+ ˆ i 1 å j=m+t+1 l (j) (n) b and j=m å j=mˆ s 2 +1 l (j) (n) j=m+t+ ˆ i 2 å j=m+t+1 l (j) (n) a , (4.32) ˇ D t =fi 1 ( ˇ T t ), ..., i m+t ( ˇ T t )g. (4.33) In this expression ˆ s 1 = k 1 t, ˆ i 1 = maxf` m+ k 1 t, 0g and ˆ s 2 = maxfm+ k 2 u, 0g and ˆ i 2 = k 2 . This procedure rejects the m+ t hypotheses with the largest log-likelihood ratios when the gaps are larger than a and b for every t with 0< t< k 1 . Next we define the stopping time for a target number m of signals as T m = min min 0t<k ˇ T t , min 0<t<k ˆ T t , (4.34) D m = [ 0t<k ˇ D t [ [ 0<t<k ˆ D t . (4.35) This procedure combines ˇ T t and ˆ T t and stops as soon as any of these procedures does so, and uses the corresponding decision rule upon stopping. We now finally define the leap sum rule as stopping the first time any corre- sponding stopping time T m stops, for any target number of signals` m u. That is, for` m u, the family-wise leap sum rule is T FLS = min `mu fT m g, (4.36) D FLS = [ `mu D m . (4.37) Chapter4. Procedurestocontrol k-family-wiseerrorrates 72 4.3.1 Error control for the family-wise leap sum rule We proceed to show how to find the thresholds a and b to guarantee the desired error control for the family-wise leap sum rule. Lemma 4.3.1. Assume the streams (1.3) are independent. Given 0< a< 1, 0< b< 1 and 0<`< u< J, if a=j log(b)j+ log m=u å m=` t=k 2 1 å t=0 C ˇ s 2 mt C ˇ i 2 Jm + t=k 1 1 å t=1 C ˆ s 2 m C ˆ i 2 Jm !! (4.38) b=j log(a)j+ log m=u å m=` t=k 2 1 å t=0 C ˇ s 1 mt C ˇ i 1 Jm + t=k 1 1 å t=1 C ˆ s 1 m C ˆ i 1 Jmt !! , (4.39) then(T FLS , D FLS )2D a,b (P `,u ). Proof. Fix t with 0 t < k 2 and m with ` m u. If the procedure re- jects the m t hypotheses with the largest log-likelihood ratios, with ˇ s 1 = k 1 false positives, the minimal number of false negatives besides the first t is ˇ i 1 = maxfl m+ k 1 , 0g. Since the number of hypotheses that get rejected is m t and the number of false positives is at least k 1 , then the number of true positives is at most m t k 1 . We know the total number of positives is at least l, thus the number of false negatives is at least maxfl m+ k 1 + t, 0g. Thus the minimal number of false negatives besides the first t is ˇ i 1 = maxfl m+ k 1 , 0g. Fix A [J] withjAj = m, then there exists a set of false negatives B 1 A c (recalling that the latter denotes the complement of A) withjB 1 j = t+ ˇ i 1 and a set of false positives B 2 A withjB 2 j = ˇ s 1 . Then besides the first t hypotheses that get accepted with the largest log-likelihood ratios, there exists M B 1 with jMj= ˇ i 1 . Then fj ˇ D t n Aj k 1 g [ B 2 2 A and M2 A c andjB 2 j= ˇ s 1 andjMj= ˇ i 1 . Chapter4. Procedurestocontrol k-family-wiseerrorrates 73 The union is over all possible selections of set M and B 2 . With a change of measure P A ! P C with C =(An B 2 )[ M, from (1.9) and Wald’s likelihood ratio identity it follows that P A (j ˇ D t n Aj k 1 ) m t ˇ s 1 J m ˇ i 1 E C [expf å j2B 2 l j å j2M l j g] m t ˇ s 1 J m ˇ i 1 e b . (4.40) Since for any pair of log likelihood ratios whose indices are in B 2 and M, respec- tively, the gap between them is larger than b. The inequality holds for any t with 0 t< k 2 . Similarly, for 0< t< k 1 : P A (j ˆ D t n Aj k 1 ) m ˆ s 1 J m t ˆ i 1 e b . (4.41) Then we have P A (fjD m n Aj k 1 g) t=k 2 1 å t=0 m t ˇ s 1 J m ˇ i 1 + t=k 1 1 å t=1 m ˆ s 1 J m t ˆ i 1 ! e b . (4.42) This last inequality holds for any m with` m u. Then finally, P A (fjD FLS n Aj k 1 g) m=u å m=` P A (fjD m n Aj k 1 g) = m=u å m=` t=k 2 1 å t=0 m t ˇ s 1 J m ˇ i 1 + t=k 1 1 å t=1 m ˆ s 1 J m t ˆ i 1 ! e b a. (4.43) Chapter4. Procedurestocontrol k-family-wiseerrorrates 74 Similarly, we have P A (fjAn D FLS j k 2 g) m=u å m=` P A (fjAn D m j k 2 g) = m=u å m=` t=k 2 1 å t=0 m t ˇ s 2 J m ˇ i 2 + t=k 1 1 å t=1 m ˆ s 2 J m ˆ i 2 ! e a b (4.44) 4.3.2 Asymptotic optimality of the family-wise leap sum rule In this section show that the family-wise leap sum rule is asymptotically op- timal in the sense of asymptotically minimizing the expected sample size as a andb go to 0 among all procedures such that the probability of family-wise type I error is no larger than a and family-wise type II error metric is no larger than b. For this we first need to obtain an asymptotic lower bound on the optimal expected sample size for procedures in the classD a,b (P `,u ), recall that D a,b (P m )=f(T, D) : P A (jDn Aj k 1 ) a and P A (jAn Dj k 2 ) b and every A2P m g. For a given signal set A, we define the following constants in which the infor- mation numbers are with respect to P A . Recall that in the following equation ˇ s 1 = k 1 , ˇ i 1 = maxf` m+ k 1 , 0g, ˇ s 2 = maxfm+ k 2 t u, 0g, ˇ i 2 = k 2 t and Chapter4. Procedurestocontrol k-family-wiseerrorrates 75 ˆ s 1 = k 1 t, ˆ i 1 = maxf` m+ k 1 t, 0g, ˆ s 2 = maxfm+ k 2 u, 0g, ˆ i 2 = k 2 . ˇ h t,1 (A)= j=ˇ s 1 å j=1 I (t+j) 1 + j= ˇ i 1 å j=1 I (j) 0 , (4.45) ˆ h t,1 (A)= j=ˆ s 1 å j=1 I (j) 1 + j= ˆ i 1 å j=1 I (t+j) 0 . (4.46) ˇ h t,2 (A)= j=ˇ s 2 å j=1 I (t+j) 1 + j= ˇ i 2 å j=1 I (j) 0 , (4.47) ˆ h t,2 (A)= j=ˆ s 2 å j=1 I (j) 1 + j= ˆ i 2 å j=1 I (t+j) 0 . (4.48) Givena,b2(0, 1), let ˆ L t (A)= j log(a)j ˆ h t,1 _ j log(b)j ˆ h t,2 , (4.49) ˇ L t (A)= j log(a)j ˇ h t,1 _ j log(b)j ˇ h t,2 . (4.50) Finally, define L m (A)= min min 0t<k 2 ˇ L t (A), min 0<t<k 1 ˆ L t (A) . (4.51) In order to prove the lower bound of the admissible classD a,b (P `,u ), we used a similar idea as the proof of Theorem 3.2.1. Lemma 4.3.2. Fix m2 [J 1]. Let A, B [J] be such thatjAj = m,jBn Aj < k 1 , andjAn Bj < k 2 . Then there exists B 1 [J] and B 2 [J] with `jB 1 j u, `jB 2 j u, and satisfying the following. (i) IfjBj k 1 andjB C j k 2 , then there exist B 1 , B 2 [J] such that (1)jBn B 1 j= k 1 andjB 2 n Bj= k 2 , (2) j log(a)j I A,B 1 _ log(b)j I A,B 2 L m (A). Chapter4. Procedurestocontrol k-family-wiseerrorrates 76 (ii) IfjBj< k 1 , then there exists B 2 [J] such that (1)jB 2 n Bj= k 2 , (2) j log(b)j I A,B 2 L m (A). (iii) IfjB C j< k 2 , then there exists B 1 [J] such that (1)jBn B 1 j= k 2 , (2) j log(a)j I A,B 1 L m (A). Proof. For (i), let a = jA/Bj and b = jB/Aj, let C A\ B be the indices of the smallest s 2 = maxfm+ k 2 a u, 0g information numbers in A\ B. Similarly, let D(A[ B) c be the indices of the smallest i 2 = k 2 a information numbers in (A[ B) c . Then set B 2 = (An B)[((A\ B)n C)[ D. We see that B 2 n B=(An B)[ D, thusjB 2 n Bj= a+ k 2 a= k 2 ,jB 2 j= m s 2 + k 2 a u, andjB 2 j`. It follows that I A,B 2 = I C + I D . Similarly, we can construct B 1 as follows: let C A\ B be the indices of the smallest s 1 = k 1 b information numbers in A\ B. Similarly, let D (A[ B) c be the indices of the smallest i 1 = maxf` m+ k 1 b, 0g information numbers in(A[ B) c . Then set B 1 =(An B)[((A\ B)n C)[ D. We can see that Bn B 1 = (Bn A)[ C, thusjBn B 1 j= b+ s 1 = k 1 andjB 1 j= m s 1 + i 1 ` andjB 1 j u. It follows that I A,B 1 = I C + I D . If a b, let t= a b. Then ˇ h ab,1 (A)= j=ab+ˇ s 1 å j=ab+1 I (j) 1 + j= ˇ i 1 å j=1 I (j) 0 = j=ab+k 1 å j=ab+1 I (j) 1 + j=maxf`m+k 1 ,0g å j=1 I (j) 0 j=a+k 1 b å j=a+1 I (j) 1 + j=b+maxf`m+k 1 b,0g å j=b+1 I (j) 0 I A,B 1 . (4.52) Chapter4. Procedurestocontrol k-family-wiseerrorrates 77 Similarly, we can prove that ˇ h ab,2 (A) I A,B 2 . (4.53) Thus j log(a)j I A,B 1 _ j log(b)j I A,B 2 ˇ L ab (A) (4.54) If a< b, then with t= b a we can argue similarly that j log(a)j I A,B 1 _ j log(b)j I A,B 2 ˆ L ba (A). Thus j log(a)j I A,B 1 _ j log(b)j I A,B 2 L m (A). Thus condition (i).(2) is satisfied. The proof of (ii) and (iii) are similar. With Lemma 4.3.2 established we are now ready to prove Theorem 4.3.1 which establishes a lower bound on the expected sample size of the procedures in the admissible classD a,b (P `,u ). Theorem 4.3.1. Suppose the data streams are independent and the assumption of SLLN (1.6) holds. Then for any A[J] withjAj= m, we have that inf (T,D)2D a,b (P `,u ) E A [T] L m (A)(1+ o(1)) asa,b! 0. Proof. The proof is a modification of the proof of Song and Fellouris (2016, Theorem 4.3) by restricting the set with signals A to bejAj= m. In their proof A is arbitrary, but here withjAj = m, when we choose B 1 and B 2 in Lemma 4.3.2, we require`jB 1 j u and`jB 2 j u, and the resulting constant becomes Chapter4. Procedurestocontrol k-family-wiseerrorrates 78 L m (A). The other steps in the proof are largely the same as in Song-Fellouris and so we omit them here. The following lemma establishes an asymptotic upper bound for the ex- pected sample size of the family-wise leap rule. Lemma 4.3.3. Assume that SLLN (1.6) holds. Then for any A[J] withjAj= m, as a, b!¥ and bj log(a)j and aj log(b)j, we have E A [ ˇ T t ] b(1+ o(1)) ˇ h t,1 _ a(1+ o(1)) ˇ h t,2 (A) for every 0 t< k, and E A [ ˆ T t ] b(1+ o(1)) ˆ h t,1 _ a(1+ o(1)) ˆ h t,2 (A) for every 0< t< k. Proof. Fix A and define the following classes of subsets: M 1 =fB A, C A C :jBj= ˆ s 1 ,jCj= ˆ i 1 , I i 0 I t+1 0 ,8i2 Cg, M 2 =fB A, C A C :jBj= ˆ s 2 ,jCj= ˆ i 2 , I i 0 I t+1 0 ,8i2 Cg. Define T 0 t = inf n 1 : min B,C2M 1 å j2B l j (n) å j2C l j (n) b, min B,C2M 2 å j2B l j (n) å j2C l j (n) a . (4.55) Chapter4. Procedurestocontrol k-family-wiseerrorrates 79 Similar to the proof of Lemma 4.2.3, we have ˆ T t ˆ T 0 t a.s. Thus, we have E A [ ˆ T t ] b(1+ o(1)) å j= ˆ i 1 j=1 I (j+t) 0 +å j=ˆ s 1 j=1 I (j) 1 _ a(1+ o(1)) å j= ˆ i 2 j=1 I (j+t) 0 +å j=ˆ s 2 j=1 I (j) 1 = b(1+ o(1)) ˆ h t,1 (A) _ a(1+ o(1)) ˆ h t,2 (A) . (4.56) Similarly, E A [ ˇ T t ] b(1+ o(1)) ˇ h t,1 (A) _ a(1+ o(1)) ˇ h t,2 (A) . (4.57) Thus, since bj log(a)j and aj log(b)j we have E A [T FLS ] E A [T m ]= min min 0t<k 2 ˇ T t min 0<t<k 1 ˆ T t L m (A)(1+ o(1)). (4.58) Using Theorem 4.3.1 and Lemma 4.3.3, we are now ready to establish the asymptotic optimality of the family-wise leap sum rule. Theorem 4.3.2. Assume the data streams are independent and the assumption of SLLN (1.6) holds. Then for any A with`jAj u and A[J], we have that asa,b! 0, E A [T FLS ] L m (A)(1+ o(1)) inf (T,D)2D a,b (P `,u ) E A [T] ifjAj= m. Proof. The a, b in Lemma 4.3.1 satisfy bj log(a)j and aj log(b)j. And from (4.58), E A [T FL ] L m (A)(1+ o(1)). (4.59) And from Theorem 4.3.1 inf (T,D)2D a,b (P `,u ) E A [T] L m (A)(1+ o(1)). (4.60) Chapter4. Procedurestocontrol k-family-wiseerrorrates 80 The lower bound is same order as the upper bound, which proves the family- wise leap sum rule attains the asymptotic lower bound. 4.4 Effects of prior information In this section we compare the effects of the varying degrees of prior infor- mation in the procedures proposed above for controlling the k-family-wise error metric. This is done first through Theorem 4.4.1, which establishes that, not surprisingly, the limiting expected sample sizes decrease as more precise prior information is considered. Next, for a more detailed, finite-sample comparison, we present a simulation study of the procedures in Table 4.1. Theorem 4.4.1. Assume the data streams 1.3 are independent and the assumption (1.6) of SLLN holds. Fix arbitrary A [J] and let m =jAj and 0 ` m u J. For a2 (0, 1) and b2 (0, 1), we consider the following procedures: Let T F L denote the family-wise leap rule with parametersa and m; let T FLS denote the family-wise leap sum rule with parameters a,`, and u; and let T L denote the leap rule. Then as a! 0 and b! 0, the first order terms in the asymptotic expansions of the expected stopping time E A [T] decrease with more information about the number of signals. That is, E ¥ A [T FL ] E ¥ A [T FLS ] E ¥ A [T L ]. where E ¥ A (T) :=j logaj lim a!0 E A (T) j logaj _j logbj lim b!0 E A (T) j logbj is the first order term in the asymptotic expansion of the expected sample size of the procedure T = T(a,b). Proof. In order to prove the first inequality, since ˇ s 1 = k 1 and ˇ i 1 = maxf` m+ k 1 , 0g k 1 . ˇ s 2 = maxfm+ k 2 t u, 0g k 2 t and ˇ i 2 = k 2 t. And similarly ˆ s 1 = k 1 t and ˆ i 1 = maxfl m+ k 1 t, 0g k 1 t. ˆ s 2 = maxfm+ Chapter4. Procedurestocontrol k-family-wiseerrorrates 81 k 2 u, 0g k 2 and ˆ i 2 = k 2 . Thus we have ˇ h t,1 (A) ˇ h l,1 (A), ˇ h t,2 (A) ˇ h l,2 (A), ˆ h t,1 (A) ˆ h l,1 (A) and ˆ h t,2 (A) ˆ h l,2 (A). Then we have E A [T FL ] E A [T FLS ]. For the second inequality, since ˇ s 1 = k 1 and ˇ i 1 = maxfl m+ k 1 , 0g 0. ˇ s 2 = maxfm+ k 2 t u, 0g 0 and ˇ i 2 = k 2 t. Similarly ˆ s 1 = k 1 t and ˆ i 1 = maxfl m+ k 1 t, 0g 0. ˆ s 2 = maxfm+ k 2 u, 0g 0 and ˆ i 2 = k 2 . Thus E A [T FLS ] E A [T L ]. Next we present a simulation study of the finite-sample performance of the procedures in the theorem. We use the same normal mean setting we used in Section 2.5. We consider J = 10 independent streams X j 1 , X j 2 , . . ., j2 [J], of i.i.d. N(m, 1) data with hypotheses H j 0 : m = 0 vs. H j 1 : m = 1/2, j2 [J]. We set the number of signals in these data streams to bejAj = 5, and the identities of the signal streams is chosen arbitrarily since the procedures’ performance is invari- ant with respect to this. In this case the information numbers are I j 0 = I j 1 = 1/8. Furthermore we set a = 0.05 and b = 0.05, and for each procedure values of a and b was determined by Monte Carlo simulation such that the achieved family- wise type I and type II error are close to, but no larger than, these values. The procedures in the table are as follows. We take the number of signals to be m= 5 and consider bounds (`, u) = (0, 10),(1, 9), . . . ,(4, 6),(5, 5) on the number of signals. For the cases with 0 < ` < u < 10, the procedure T is the family-wise leap sum rule with parameters` and u. For the case with` = u = m = 5, it is the family-wise leap rule with parameters m. In the case with`= 0 and u = 10, it is the leap rule. For each of these procedures, the expected stopping time ET and achieved family-wise type I and type II errors, estimated from 10,000 Monte Carlo replications, are given in the table; standard errors are given in parenthe- ses. The first order terms in the asymptotic expansions (computed exactly) as in Equations (4.18) and (4.51) are also computed accordingly and given in the table. Chapter4. Procedurestocontrol k-family-wiseerrorrates 82 l u a b E[T] Computed first order term family-wise type I error family-wise type II error 0 10 -2.7 2.7 64.40 (5.31) 24 0.050 (0.002) 0.050 (0.002) 1 9 -2.7 2.7 64.64 (5.25) 24 0.049 (0.002) 0.043 (0.002) 2 8 -2.7 2.7 64.35 (5.46) 24 0.048 (0.002) 0.047 (0.002) 3 7 -2.7 2.7 64.44 (5.35) 24 0.048 (0.002) 0.047 (0.002) 4 6 -2.7 2.7 64.15 (5.38) 24 0.050 (0.002) 0.048 (0.002) 5 5 -3.6 3.6 42.54 (2.73) 12 0.049 (0.002) 0.049 (0.002) TABLE 4.1: Comparison of expected stopping times for procedures with varying degrees of prior information, as described in Sec- tion 4.4 with J = 10 data streams. From the table, we can see the expected sample size E[T] decrease as the bounds(`, u) get tighter around m and there is a substantial decrease in sample size for ` = u = m = 5 case. This behavior of the expected sample size is mirrored by the first order asymptotic terms given in the table. Thus, there is a relatively large “cost” (in terms of average sample size) to the family-wise leap sum rule when the true number of signals is strictly between the bounds ` <jAj < u, because this procedure must always consider the possibility that jAj = ` or u, versus the family-wise leap rule with m = jAj. But perhaps surprisingly, this cost is relatively insensitive to how tight these bounds`, u are around m, until` = u = m, with even the case` = m 1, u = m+ 1 being not much different than cases with substantially smaller` and larger u. 83 Chapter 5 Discussion 5.1 Relationship to classification problems The problems addressed in Chapters 2, 3 and 4 may naturally be considered classification problems, in which J objects are each classified into one of two re- spective classes, i.e., the corresponding null and alternative hypotheses. This is essentially the same setting for which classification via FDR thresholding has been investigated for fixed-sample data by many authors including Abramovich et al. (2006), Bogdan, Ghosh, and Tokdar (2008), Donoho and Jin (2006), Gen- ovese and Wasserman (2002), and Neuvial and Roquain (2012). FDR, general- ized error metrics, and k-family-wise error metrics have previously been con- sidered as classifiers based on sequential data (e.g., Bartroff and Song, 2013; Bartroff, 2017) but this thesis appears to be the first time asymptotic optimal- ity has been considered and achieved for these metrics. The setup considered in Chapter 1 accommodates more specialized clas- sification problems as well. For example, slippage problems (see Draglia, Tar- takovsky, and Veeravalli, 1999; Ferguson, 1967; Mosteller, 1948) consider J in- dependent populations, of which at most one is in a non-null state, represented by the alternative hypothesis. In the fixed-sample setting, Ferguson (1967) con- sidered a slippage problem in which both the null and alternative are known. Chapter5. Discussion 84 Tartakovsky (1997) found minimax solutions for more general hypotheses, as did Draglia, Tartakovsky, and Veeravalli (1999) in a sequential Bayesian setting. This thesis can address slippage problems by taking the bounds in Sections 2.4, 3.3 and 4.3 on the number of signals to be ` = 0 and u = 1, since at most one stream has a signal, and allows the asymptotically optimal application of FDR/FNR in Chapter 2, generalized error metric in Chapter 3, or the k-family- wise error metric in Chapter 4 by applying Theorem 2.4.1, Theorem 3.3.2, or Theorem 4.3.2, respectively. Focusing on FDR for a moment, we note that its utility beyond FWE 1 is inherently limited in slippage problems because the two metrics will coincide for any procedure which chooses either ` = 0 or u = 1 signals. This is because, using the notation in Section 1.3, both the number V of falsely rejected nulls and the number R of rejected nulls can only take the values 0 or 1, hence FDR= E V R_ 1 = E V 1 = P(V = 1)= P(V 1)= FWE 1 . (5.1) However, the same does not hold for FNR and FWE 2 in this case hence, in spite of (5.1), the asymptotically optimal FDR/FNR control in this thesis is distinct from the FWE control of Song and Fellouris (2017) in slippage problems and its application here is novel, as well as any other metrics satisfying Theorem 2.4.1 as well as the metrics considered in Chapters 3 and 4. 5.2 Further extensions The problems considered above were under the assumption that the data streams for the various hypotheses are independent. An open area of research is how to relax this assumption. A completely general case may be too complex to consider initially, but special cases may offer insight. For example, suppose Chapter5. Discussion 85 each “slice” of the data X = (X 1 n , ...X J n ) at time n comprises a multivariate nor- mal vector with known co-variance matrix. Then one may be able to create an independent normal vector by using “pre-whitening” linear transformation and applying the sequential multiple testing procedure with the transformed data. A remaining difficulty with this approach is that one is then considering linear transformations of the original hypotheses. If this could be overcome then one could move on to more general cases, such as knowledge of the co-variance ma- trix up to proportionality constant, etc., or completely unknown in which it is estimated from the data. Another open area for extension is to consider higher order asymptotic opti- mality. In the results we have conducted a first-order asymptotic analysis, ignor- ing higher order terms in the approximation to the optimal performance. Thus, a natural next step would be to obtain a more precise characterization of the optimal performance through higher order terms. 86 Bibliography Abramovich, Felix et al. (2006). “Adapting to unknown sparsity by controlling the false discovery rate”. In: The Annals of Statistics 34.2, 584–653. DOI: 10.12 14/009053606000000074. Bartroff, Jay (2017). “Multiple Hypothesis Tests Controlling Generalized Error Rates for Sequential Data”. In: Statistica Sinica. DOI:10.5705/ss.202015.026 7. Bartroff, Jay and Tze Leung Lai (2010). “Multistage Tests of Multiple Hypothe- ses”. In: Communications in Statistics - Theory and Methods 39.8-9, 1597–1607. DOI:10.1080/03610920802592852. Bartroff, Jay and Jinlin Song (2013). “Sequential tests of multiple hypotheses con- trolling false discovery and nondiscovery rates.” In: Under review. — (2014). “Sequential tests of multiple hypotheses controlling type I and II fam- ilywise error rates”. In: Journal of Statistical Planning and Inference 153, 100–114. DOI:10.1016/j.jspi.2014.05.010. Benjamini, Yoav and Yosef Hochberg (1995). “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing”. In: Journal of the Royal Statistical Society: Series B (Methodological) 57.1, 289–300. DOI: 10.11 11/j.2517-6161.1995.tb02031.x. Black, M. A. (2004). “A note on the adaptive control of false discovery rates”. In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 66.2, 297–304. DOI:10.1111/j.1369-7412.2003.05527.x. BIBLIOGRAPHY 87 Bogdan, Małgorzata, Jayanta K. Ghosh, and Surya T. Tokdar (2008). “A compari- son of the Benjamini-Hochberg procedure with some Bayesian rules for mul- tiple testing”. In: Beyond Parametrics in Interdisciplinary Research: Festschrift in Honor of Professor Pranab K. Sen Collections, 211–230. DOI: 10.1214/19394030 7000000158. Bonferroni, Carlo E. (1936). In: SpringerReference. DOI:10.1007/springerrefere nce_60803. Burke, Donald S. et al. (1988). “Measurement of the False Positive Rate in a Screening Program for Human Immunodeficiency Virus Infections”. In: New England Journal of Medicine 319.15, 961–964. DOI:10.1056/nejm198810133191 501. De, Shyamal K. and Michael Baron (2012). “Sequential Bonferroni Methods for Multiple Hypothesis Testing with Strong Control of Family-Wise Error Rates I and II”. In: Sequential Analysis 31.2, 238–262. DOI: 10.1080/07474946.2012 .665730. Donoho, David and Jiashun Jin (2006). “Asymptotic minimaxity of false discov- ery rate thresholding for sparse exponential data”. In: The Annals of Statistics 34.6, 2980–3018. DOI:10.1214/009053606000000920. Draglia, V .p., A.g. Tartakovsky, and V .v. Veeravalli (1999). “Multihypothesis se- quential probability ratio tests .I. Asymptotic optimality”. In: IEEE Transac- tions on Information Theory 45.7, 2448–2461. DOI:10.1109/18.796383. Ferguson, T. S. (1967). “Mathematical Statistics: A Decision Theoretic Approach.” In: The Annals of Mathematical Statistics. Frane, Andrew V (2015). “Are Per-Family Type I Error Rates Relevant in Social and Behavioral Science?” In: Journal of Modern Applied Statistical Methods 14.1, 12–23. DOI:10.22237/jmasm/1430453040. BIBLIOGRAPHY 88 Genovese, Christopher and Larry Wasserman (2002). “Operating characteristics and extensions of the false discovery rate procedure”. In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 64.3, 499–517. DOI: 10.111 1/1467-9868.00347. Govindarajulu, Zakkula (1987). The sequential statistical analysis of hypothesis test- ing, point and interval estimation, and decision theory. American sciences Press. He, Xinrui and Jay Bartroff (2019). “Asymptotically optimal sequential FDR and pFDR control with (or without) prior information on the number of signals”. In: Under review arXiv:1905.00177. Hochberg, Yosef (1988). “A Sharper Bonferroni Procedure for Multiple Tests of Significance”. In: Biometrika 75.4, p. 800. DOI:10.2307/2336325. Jennison, Christopher and Bruce W. Turnbull (1993). “Sequential Equivalence Testing and Repeated Confidence Intervals, with Applications to Normal and Binary Responses”. In: Biometrics 49.1, p. 31. DOI:10.2307/2532600. Jiang, H. and J. Salzman (2012). “Statistical properties of an early stopping rule for resampling-based multiple testing”. In: Biometrika 99.4, 973–980. DOI: 10 .1093/biomet/ass051. Keselman, H. J. (2015). “Per Family or Familywise Type I Error Control: "Eether, Eyether, Neether, Nyther, Lets Call the Whole Thing Off!"” In: Journal of Mod- ern Applied Statistical Methods 14.1, 24–37. DOI:10.22237/jmasm/1430453100. Malloy, Matthew L. (2014). “Sequential Testing for Sparse Recovery”. In: Practi- cal Applications of Sparse Modeling. DOI:10.7551/mitpress/9333.003.0009. Mei, Yajun (2008). “Asymptotic Optimality Theory for Decentralized Sequential Hypothesis Testing in Sensor Networks”. In: IEEE Transactions on Information Theory 54.5, 2072–2089. DOI:10.1109/tit.2008.920217. BIBLIOGRAPHY 89 Mosteller, Frederick (1948). “A k-Sample Slippage Test for an Extreme Popula- tion”. In: The Annals of Mathematical Statistics 19.1, 58–65. DOI: 10.1214/aoms /1177730290. Neuvial, Pierre and Etienne Roquain (2012). “On false discovery rate threshold- ing for classification under sparsity”. In: The Annals of Statistics 40.5, 2572–2600. DOI:10.1214/12-aos1042. Paulson, Edward (1964). “A Sequential Procedure for Selecting the Population with the Largest Mean from k Normal Populations”. In: The Annals of Mathe- matical Statistics 35.1, 174–180. DOI:10.1214/aoms/1177703739. Siegmund, D. (1993). “A Sequential Clinical Trial for Comparing Three Treat- ments”. In: The Annals of Statistics 21.1, 464–483. DOI: 10.1214/aos/1176349 037. Siegmund, David (2010). Sequential analysis: tests and confidence intervals. Springer- Verlag. Sobel, Milton and Abraham Wald (1949). “A Sequential Decision Procedure for Choosing One of Three Hypotheses Concerning the Unknown Mean of a Normal Distribution”. In: The Annals of Mathematical Statistics 20.4, 502–522. DOI:10.1214/aoms/1177729944. Song, Yanglei and Georgios Fellouris (2016). “Sequential multiple testing with generalized error control: An asymptotic optimality theory”. In: The Annals of Statistics 47.3, 1776–1803. DOI:10.1214/18-aos1737. — (2017). “Asymptotically optimal, sequential, multiple testing procedures with prior information on the number of signals”. In: Electronic Journal of Statistics 11.1, 338–363. DOI:10.1214/17-ejs1223. Storey, John D. (2002). “A direct approach to false discovery rates”. In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 64.3, 479–498. DOI: 10.1111/1467-9868.00346. BIBLIOGRAPHY 90 Tartakovsky, A. G. (1997). “Minimax invariant regret solution to the N-sample slippage problem.” In: Mathematical Methods of Statistics. Tartakovsky, Alexander, I. V . Nikiforov, and M. Basseville (2015). Sequential anal- ysis: hypothesis testing and changepoint detection. Taylor and Francis. Wald, A. (1945). “Sequential Tests of Statistical Hypotheses”. In: The Annals of Mathematical Statistics 16.2, 117–186. DOI:10.1214/aoms/1177731118. Wegman, Edward J and David J Marchette (2003). “On Some Techniques for Streaming Data: A Case Study of Internet Packet Headers”. In: Journal of Com- putational and Graphical Statistics 12.4, 893–914. DOI:10.1198/1061860032625.

Abstract (if available)

Abstract We investigate asymptotically optimal multiple testing procedures for streams of sequential data in the context of prior information on the number of false null hypotheses (“signals”). We consider different pairs of error metrics (FDR/FNR, pFDR/pFNR, generalized error and k family-wise error) and for each of these metrics, we consider sequential multiple testing procedures that achieve asymptotic optimality in the sense of minimizing the expected sample size to first order as the type 1 and 2 versions of these error metrics approach 0 at arbitrary rates. This is done in the setting of testing simple null/alternative hypothesis pairs, and where there is some degree (e.g., zero) of prior information on the number of false null hypothesis (“signals”). In particular, we show that the “gap” and “gap-intersection” procedures, recently proposed and shown by Song and Fellouris (2017) to be asymptotically optimal for controlling type 1 and 2 family-wise error rates (FWEs), are also asymptotically optimal for controlling FDR/FNR when the critical values are appropriately adjusted. Generalizing this result, we show that these procedures, again with appropriately adjusted critical values, are asymptotically optimal for controlling any multiple testing error metric that is bounded between multiples of FWE in a certain sense. This enlarged class of metrics includes FDR/FNR but also pFDR/pFNR, the per comparison error rate, and other metrics. We also generalize the “sum"" and “leap” procedures, proposed by Song and Fellouris (2016) for controlling generalized error and k-family-wise error with prior information on the number of signals.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Sequential testing of multiple hypotheses

PDF

Sequential testing of multiple hypotheses with FDR control

PDF

Large-scale multiple hypothesis testing and simultaneous inference: compound decision theory and data driven procedures

PDF

Finite sample bounds in group sequential analysis via Stein's method

PDF

Large scale inference with structural information

PDF

Multi-population optimal change-point detection

PDF

Topics in selective inference and replicability analysis

PDF

Equilibrium model of limit order book and optimal execution problem

PDF

Reproducible large-scale inference in high-dimensional nonlinear models

PDF

Statistical insights into deep learning and flexible causal inference

PDF

Large-scale inference in multiple Gaussian graphical models

PDF

Stein's method and its applications in strong embeddings and Dickman approximations

PDF

Model selection principles and false discovery rate control

PDF

High dimensional estimation and inference with side information

PDF

Evaluating the effects of testing framework and annotation updates on gene ontology enrichment analysis

PDF

Shrinkage methods for big and complex data analysis

PDF

Essays on nonparametric and finite-sample econometrics

PDF

Compression of signal on graphs with the application to image and video coding

PDF

Sequential decision-making for sensing, communication and strategic interactions

PDF

Three essays on econometrics

Asset Metadata

Creator He, Xinrui (author)

Core Title Asymptotically optimal sequential multiple testing with (or without) prior information on the number of signals

School College of Letters, Arts and Sciences

Degree Doctor of Philosophy

Degree Program Applied Mathematics

Publication Date 12/06/2019

Defense Date 09/06/2019

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag FDR,number of signals,OAI-PMH Harvest,sequential multiple testing

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Bartroff, Jay (committee chair), Lototsky, Sergey (committee member), Lv, Jinchi (committee member)

Creator Email hexinrui123@gmail.com,xinruihe@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-248499

Unique identifier UC11675318

Identifier etd-HeXinrui-8019.pdf (filename),usctheses-c89-248499 (legacy record id)

Legacy Identifier etd-HeXinrui-8019.pdf

Dmrecord 248499

Document Type Dissertation

Rights He, Xinrui

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA