Another prime example of how far MNRAS still has to go if it’s to learn how to properly review statistical papers from last week’s astro ph: “Recent multi-kiloton impact events: are they truly random?” (http://arxiv.org/pdf/1409.0452.pdf).
In this paper the authors (de la Fuente Marcos x 2) seek to test the hypothesis that Earth-striking meteoroids are “completely random, and that all the impacts must be interpreted as uncorrelated evens distributed according to Poisson statistics. If this is correct, their impact dates must be uniformly spread throughout the year, and their impact coordinates must be evenly scattered on the surface of our planet.“ To answer this question the authors examine the calendar dates for a compilation of impact dates for 33 large impacts occurring between 2000 and 2014. As a preliminary investigation they note, “Thirteen impacts have been recorded on the first part of the year and 20 on the second, a 21 percent relative difference with respect to the evenly-distributed scenario.” So we are very much given the impression that the authors are aiming to test the hypothesis of a uniform distribution over calendar days. The usual way this would be done is via the K-S test of the empirical CDF of impact calendar days against the benchmark CDF of the uniform; modulo the technical caveat that the K-S test is designed for continuous data whereas the impact times have been binned into discrete days. Applying this test the authors could not have rejected the null as the p-value returned is 0.13. Likewise Kuiper’s test (designed for invariance under cyclic transformations; favourable for testing variations by day of the year; thanks Aaron) also refused to reject the null, giving a p-value of >0.15.
Instead the authors risk accusations of post-hoc-ery by deciding to use as test statistic the number of same day calendar pairs, being for this sample a total of 4, which they authors note has a 4.05% chance of occurring with 33 draws from the uniform distribution over calendar days. Moreover, they note, there are a number of other similar statistics one can observed in the data: e.g. the number of pairs from events within one day of each other (9 with probability under the null of 1.9%); the same within two days (14 / 0.93%), etc. From which they conclude, “The number of same-day (or nearly same-day) coincidences is simply too high to be the result of chance alone. It is statistically obvious that the impact events in Table A1 are not uniformly distributed in time.” However, apart from the unusual choice of test statistic (which suggests an alternative hypothesis of inauspicious days for meteoroid strikes perhaps!) the authors haven’t actually performed a statistical hypothesis test here. Although the same day probability is 4.05% the corresponding one-sided p-value for probability of 4 *or more* same-day pairs is 0.0583, which could not be rejected at a sensible significance: say 0.05 or 0.01.
It has been suggested that the expected turn-around time (~3 weeks) for reviewers assigned by MNRAS might be too short to allow for a thorough statistical checking, but I don’t think this is the case [this short paper would not have taken more than a half day to read through carefully]. My theory is that the MNRAS editorial team simply has its head in the sand with regard to the problem of bad astrostatistics, and it won’t be until a proper headline result falls over from bad stats that they’ll finally get their act together. My suggestion would be for any paper having a result depending wholly, or in large part, on statistical analysis should be automatically assigned for a once-over by a recognised statistician.