There’s another of the crazy papers on astro ph today about the handedness of spiral galaxies; this one by serial offender, Lior Shamir ( http://arxiv.org/pdf/1310.7485.pdf ). The author claims to show that there is a significant difference between the colours of clockwise spinning and anti-clockwise spinning spiral galaxies, with a probability that the difference arises through chance of just 1.9%. If we look inside the manuscript we find that the 1.9% figure is the p-value from an unpaired t-test for the difference between two sample means from populations with unknown variances. However, there were four color differences tested (u-g, g-r, r-i, i-z) and the other three had quite typical p-values of 0.26, 0.27 and 0.69. So we should probably look at the probability of obtaining at least one p-value less than or equal to 0.019 in four tries with the given sample sizes of 64399 and 63215 galaxies, respectively. A quick calculation in R gives for the example of all colours being drawn from standard normals a result of 7.8%. Not so impressive is it?!
When people complain that Bayesian model selection is too difficult I tend to refer them to examples like these where the Frequentist method, though easy to apply, has a tendency to lead us up the garden path.