A mathematician reported the results from a particular experiment to the researcher who conducted it. The report states that on one specific part of the experiment, a statistical test result yielded a p-value of 0.21. Based on this p-value, what should the researcher conclude? The test was not statistically significant because 2 × 0.21 = 0.42, which is less than 0.5.
The test was statistically significant because a p-value of 0.21 is greater than a significance level of 0.05.
The test was not statistically significant because if the null hypothesis is true, one could expect to get a test statistic at least as extreme as that observed 79% of the time.
The test was not statistically significant because if the null hypothesis is true, one could expect to get a test statistic at least as extreme as that observed 21% of the time.
The test was statistically significant because p = 1 − 0.21 = 0.79, which is greater than a significance level of 0.05.

Step-by-step explanation:

.4 Fallacies arising from overly sensitive tests

A common complaint concerning a statistically significant result is that for any discrepancy from the null, say γ ≥ 0, however small, one can find a large enough sample size n such that a test, with high probability, will yield a statistically significant result (for any p-value one wishes).

(#4) With large enough sample size even a trivially small discrepancy from the null can be detected.

A test can be so sensitive that a statistically significant difference from H0 only warrants inferring the presence of a relatively small discrepancy γ; a large enough sample size n will render the power POW(Tα; μ1=μ0 + γ) very high. To make things worse, many assume, fallaciously, that reaching statistical significance at a given level α is more evidence against the null the larger the sample size (n). (Early reports of this fallacy among psychology researchers are in Rosenthal and Gaito, 1963). Few fallacies more vividly show confusion about significance test reasoning. A correct understanding of testing logic would have nipped this fallacy in the bud 60 years ago. Utilizing the severity assessment one sees that an α-significant difference with n1 passes μ > μ1 less severely than with n2 where n1 > n2.

For a fixed type I error probability α, increasing the sample size decreases the type II error probability (power increases). Some argue that to balance the two error probabilities, the required α level for rejection should be decreased as n increases. Such rules of thumb are too tied to the idea that tests are to be specified and then put on automatic pilot without a reflective interpretation. The error statistical philosophy recommends moving away from all such recipes. The reflective interpretation that is needed drops out from the severity requirement: increasing the sample size does increase the test's sensitivity and this shows up in the “effect size” γ that one is entitled to infer at an adequate severity level. To quickly see this, consider figure 5.

Figure 5. Severity associated with inference μ> 0.2, d(x0)=1.96, and different sample sizes n.

It portrays the severity curves for test Tα, σ= 2, n= 100, with the same outcome d(x0)=1.96, but based on different sample sizes (n=50, n=100, n=1000), indicating that: the severity for inferring μ > .2 decreases as n increases:

forn=50:SEV(μ>0.2)=.895,forn=100:SEV(μ>0.2)=.831,forn=1000:SEV(μ>0.2)=.115.

The facts underlying criticism #4 are also erroneously taken as grounding the claim:

“All nulls are false.”

This confuses the true claim that with large enough sample size, a test has power to detect any discrepancy from the null however small, with the false claim that all nulls are false.

The tendency to view tests as automatic recipes for rejection gives rise to another well-known canard:

(#5) Whether there is a statistically significant difference from the null depends on which is the null and which is the alternative.

The charge is made by considering the highly artificial case of two point hypotheses such as: μ= 0 vs. μ=.8. If the null is μ= 0 and the alternative is μ=.8 then x¯=0.4 (being 2σx from 0) “rejects” the null and declares there is evidence for .8. On the other hand if the null is μ=.8 and the alternative is μ= 0, then observing x¯=0.4 now rejects .8 and finds evidence for 0. It appears that we get a different inference depending on how we label our hypothesis