Familywise Error Rate
Contents |
may be challenged and removed. (June 2016) (Learn how and when to remove this template message) In statistics, family-wise error rate (FWER) is the probability of making one false discovery rate or more false discoveries, or type I errors, among all the hypotheses
Familywise Error Rate Calculator
when performing multiple hypotheses tests. Contents 1 History 2 Background 2.1 Classification of multiple hypothesis tests 3 familywise error rate anova Definition 4 Controlling procedures 4.1 The Bonferroni procedure 4.2 The Šidák procedure 4.3 Tukey's procedure 4.4 Holm's step-down procedure (1979) 4.5 Hochberg's step-up procedure 4.6 Dunnett's correction 4.7 Scheffé's method
Familywise Error Rate Example
4.8 Resampling procedures 5 Alternative approaches 6 References History[edit] Tukey coined the terms experimentwise error rate and "error rate per-experiment" to indicate error rates that the researcher could use as a control level in a multiple hypothesis experiment.[citation needed] Background[edit] Within the statistical framework, there are several definitions for the term "family": Hochberg & Tamhane defined "family" in 1987 familywise error rate bonferroni as "any collection of inferences for which it is meaningful to take into account some combined measure of error".[1][pageneeded] According to Cox in 1982, a set of inferences should be regarded a family:[citation needed] To take into account the selection effect due to data dredging To ensure simultaneous correctness of a set of inferences as to guarantee a correct overall decision To summarize, a family could best be defined by the potential selective inference that is being faced: A family is the smallest set of items of inference in an analysis, interchangeable about their meaning for the goal of research, from which selection of results for action, presentation or highlighting could be made (Yoav Benjamini).[citation needed] Classification of multiple hypothesis tests[edit] Main article: Classification of multiple hypothesis tests The following table defines various errors committed when testing multiple null hypotheses. Suppose we have a number m of multiple null hypotheses, denoted by: H1,H2,...,Hm. Using a statistical test, we reject the null hypothesis if the test is declared significant. We do not reject the null h
or FWER. It is easy to show that if you declare tests significant for \(p < \alpha\) then FWER ≤ \(min(m_0\alpha,1)\). The most commonly used method which controls FWER at level \(\alpha\) is called Bonferroni's method. It rejects the null hypothesis
Per Comparison Error Rate
when \(p < \alpha / m\). (It would be better to use \(m_0\) but we don't
Type I Error
know what it is - more on that later.) The Bonferroni method is guaranteed to control FWER, but it has a big problem. experimentwise error rate It greatly reduces your power to detect real differences. For example, suppose the effect size is 2 and you are doing a t-test, rejecting for p < 0.05. With 10 observations per group, the power is 99%. Now https://en.wikipedia.org/wiki/Family-wise_error_rate suppose you have 1000 tests, and use the Bonferroni method. That means that to reject, we need p < 0.00005. The power is now only 29%. If you have 10 thousand tests (which is small for genomics studies) the power is only 10%. Sometimes the "Bonferroni-adjusted p-values are reported". They are just: \(p_b=min(mp,1)\). Another simple more powerful but less popular method uses the sorted p-values: \[p_{(1)}\leq p_{(2)} \leq \cdots \leq p_{(m)}\] Holmes showed that the FWER https://onlinecourses.science.psu.edu/stat555/node/58 is controlled with the following algorithm: Compare \(p_{(i)}\) with \(\alpha / (m-i+1)\). Starting from i = 1, reject until \(p_{(i)}\) is greater. The most significant test must therefore pass the Bonferroni criterion. However, if it is significant, the next most significant is tested at a less stringent level. Heuristically, after rejecting the most significant test, we conclude the \(m_0 \leq m-1\) and use \(m-1\) for the next correction, and so on sequentially. The Holmes method is more powerful than the Bonferroni method, but it is still not very powerful. We can also compute "Holmes-adjusted p-values" \(p_{h(i)} = min((m-i+1)p_{(i)},1)\). ‹ 4.1 - Mistakes in Statistical Testing up 4.3 -1995 - Two Huge Steps for Biological Inference › Printer-friendly version Navigation Start Here! Welcome to STAT 555! Faculty login (PSU Access Account) Lessons Lesson 1: Introduction to Cell Biology Lesson 2: Basic Statistical Inference for Bioinformatics Studies Lesson 3: Designing Bioinformatics Experiments Lesson 4: Multiple Testing4.1 - Mistakes in Statistical Testing 4.2 - Controlling Family-wise Error Rate 4.3 -1995 - Two Huge Steps for Biological Inference 4.4 - Estimating \(m_0\) (or \(\pi_0\)) 4.5 - q-Values 4.6 - Using the Histogram of p-values Lesson 5: Microarray Preprocessing Lesson 6: Statistics for Differential Expression in Microarray Studies Lesson 7: Linear Models for Differential Expression in Microarray Studies Lesson 8: Tables and Count Data Lesson 9: RNA Seq Data Lesson 10: Clustering Les
describe a number of different ways of testing which means are different Before describing the tests, it is necessary to consider two different ways of thinking about error and how they are relevant to doing multiple comparisons Error Rate http://www.psych.utoronto.ca/courses/c1/chap12/chap12.html per Comparison (PC) This is simply the Type I error that we have talked about all along. So far, we have been simply setting its value at .05, a 5% chance of making an error Familywise Error Rate (FW) Often, after an ANOVA, we want to do a number of comparisons, not just one The collection of comparisons we do is described as the "family" The familywise error rate is the probability that at least one of these comparisons will include error rate a type I error Assuming that a ¢ is the per comparison error rate, then: The per comparison error: a = a ¢ but, the familywise error: a = 1 - (1-a ¢ )c Thus, if we do two comparisons, but keep a ¢ at 0.05, the FWerror will really be: a = 1 - (1 - 0.05)2 =1 - (0.95)2 = 1 - 0.9025 = 0.0975 Thus, there is almost a 10% chance of one of the comparisons being significant familywise error rate when we do two comparisons, even when the nulls are true. The basic problem then, is that if we are doing many comparisons, we want to somehow control our familywise error so that we don’t end up concluding that differences are there, when they really are not The various tests we will talk about differ in terms of how they do this They will also be categorized as being either "A priori" or "post hoc" A priori: A priori tests are comparisons that the experimenter clearly intended to test before collecting any data Post hoc: Post hoc tests are comparisons the experimenter has decided to test after collecting the data, looking at the means, and noting which means "seem" different. The probability of making a type I error is smaller for A priori tests because, when doing post hoc tests, you are essentially doing all possible comparisons before deciding which to test in a formal statistical manner Steve: Significant F issue An example for context See page 351 for a very complete description of the Morphine Tolerance study .. Seigel (1975) Highlights: paw lick latency as a measure of pain resistance tolerance to morphine develops quickly notion of a compensatory mechanism this mechanism very context dependent M-S M-M S-S S-M Mc - M 3 2 14 29 24 5 12 6 20 26 1 13 12 36 40 8 6 4 21 32 1 1
be down. Please try the request again. Your cache administrator is webmaster. Generated Sat, 15 Oct 2016 13:12:02 GMT by s_wx1094 (squid/3.5.20)