Anonymous:I. Multiple testingTo remedy the problem occurred from multiple testing, we have learned Bonferroni method, Bonferroni-Holm method, and Benjamin and Hochberg false discovery rate. Bonferroni correction multiplies the number of testing to each p-value, and Bonferroni-Holm multiplies alpha/k and alpha/(k-1). Thus, I think Bonferroni is more stringent because the high p-value is applied to all the k. However, I am not able to position of false discovery rate in terms of stringency. Is the stringency of FDR located between Bonferroni and Bonferroni-Holm method, or the highest or the lowest? Also, how about we do not know the number of testing when we initiate experiments? For example, suppose we make repeatedly ANOVA test for some experiments. First, I expect 10 time experiments, and make 10 time experiment as expected. At this time I apply Bonferroni-Holm method with alpha/k and alpha/k-1. Then, we make another 10 times experiments. At this time, do I need to recalculate adjusted alpha again for the first experiments? In addition, where get the validation of this correction of p-value. I mean we can use any number to increate p-value only if p-value is lower than the original p-value. Is there any optimal p-value for multiple-testing?
" Bonferroni-Holm multiplies alpha/k and alpha/(k-1)"
Note: The pattern continues: alpha/(k-2), alpha/(k-3), ...
"Is the stringency of FDR located between Bonferroni and Bonferroni-Holm method, or the highest or the lowest? "
We haven't discussed it yet!
"Then, we make another 10 times experiments. At this time, do I need to recalculate adjusted alpha again for the first experiments? "
No, you don't have to do this. Multiplicity is something to understand, mainly, and manage where possible. It cannot be completely controlled over your whole lifetime! You don;t have to use Bonferroni, but you should understand the ratrionale for using it, and the multiplicity problem that it addresses. In particular, you should understand at a very fundamental level that ".05" means "5 % of the time".
As far as an optimal p-value for multiple testing, that is beyond the subject of our course. "Optimality" requires assumptions; different assumptions lead to different methods. Assumptions, and their reasonableness, are more important than anything else.
90 90 100 80
II. Hypothesis testingIn the hypothesis test, when we reject a null hypothesis at 5% alpha, we can think that if we make 1,000 times sampling, the probability that a true mean is not located between certain intervals is 95%. To support this argument, we need to test the null hypothesis each time whether we reject it. However, Bonferroni method presents that we need to more conservative for multiple sampling and recommend an adjusted alpha, 5%/1,000 rather than 5%. If so, I think the assumption in the hypothesis test has a problem. If we need to increase alpha lever for multiple sampling, how we can make 95% confidence for a hypothesis test? Also, is there any threshold of the number of testing for applying Bonferroni method? At how many the number of hypothesis test we need to apply Bonferroni method?
"If we need to increase alpha lever for multiple sampling, how we can make 95% confidence for a hypothesis test?"
The Bonferroni confidence interval also uses alpha/k. So there is no contradiction. Or do I not understand the question?
"Also, is there any threshold of the number of testing for applying Bonferroni method? At how many the number of hypothesis test we need to apply Bonferroni method?"
In genetics, they sometimes use it when there are 1000's of tests, and they like it! That is because most of the genetic effects truly are null, so with 1000's of tests, you need a more stringent threshhold to determine true significance. The quesiton really boils down to whether or not you want to control the FWER, not how many hypotheses there are.
What is the "integration" aspect of this question? Please look at the syllabus every week before posting your questions, and review the criteria for "general" and "specific" questions.
90 90 50 90