Silliness with p-values at big n

There’s another of the crazy papers on astro ph today about the handedness of spiral galaxies; this one by serial offender, Lior Shamir ( http://arxiv.org/pdf/1310.7485.pdf ).  The author claims to show that there is a significant difference between the colours of clockwise spinning and anti-clockwise spinning spiral galaxies, with a probability that the difference arises through chance of just 1.9%. If we look inside the manuscript we find that the 1.9% figure is the p-value from an unpaired t-test for the difference between two sample means from populations with unknown variances.  However, there were four color differences tested (u-g, g-r, r-i, i-z) and the other three had quite typical p-values of 0.26, 0.27 and 0.69.  So we should probably look at the probability of obtaining at least one p-value less than or equal to 0.019 in four tries with the given sample sizes of 64399 and 63215 galaxies, respectively.  A quick calculation in R gives for the example of all colours being drawn from standard normals a result of 7.8%.  Not so impressive is it?! 

When people complain that Bayesian model selection is too difficult I tend to refer them to examples like these where the Frequentist method, though easy to apply, has a tendency to lead us up the garden path.

Advertisements
This entry was posted in Astronomy, Astrostatistics, Bad Science, Rants. Bookmark the permalink.

5 Responses to Silliness with p-values at big n

  1. lukebarnes says:

    I was wondering about that paper …

    • I think this sort of example could serve as an adjunct to your “ten reasons to love Bayes theory” article: e.g. “the ten most common face-palms of Frequentist statistics” 🙂

  2. Sam Clifford says:

    This is the sort of mistake I’d expect to see MAB101 students making and then being beaten over the head about.

  3. I wonder if Shamir has his data available so we might try a Bayesian comparison of the same groups, per http://www.indiana.edu/~kruschke/BEST/. Could write it up as an illustration. Anyone know?

    • I don’t think this “galaxy handedness” data is available online, but in any case at the top of my hit list for bad astrostatistics papers are some more subtle, but also more pernicious, offenders …

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s