P-values, sigmas, LRTs, and Chi-sq tests …

I’ve been thinking a lot recently about non-Bayesian hypothesis testing, in part because my reading has taken me to a number of particle physics papers in which the likelihood ratio test (LRT) is very often the tool of choice for exploring new physics.  Having digested Geyer’s (1994) formulation of Chernoff’s version of the LRT—the one (Chernoff’s) used (correctly) by particle physicists when the null model lies at a boundary parameterisation of the alternative model (e.g. Chianese et al. 2017; and see Cowan et al. 2010 for a detailed exposition)—I’ve been wondering whether there are any non-trivial cases in astronomy where the asymptotic distribution can be found via the shape of the tangent cone.  One application area might be to the detection of spectral lines.  In that context Protassov et al. (2002) highlight the problem and offer a Bayesian solution, but also give the warning: “Although some theoretical progress on the asymptotic distribution of [the test statistic] when [the parameter] is on the boundary of [the parameter space] has been made (e.g., by Chernoff (1954) and specifically for finite mixtures by Lindsay (1995)), extending such results to a realistic highly structured spectral model would require sophisticated mathematical analysis“.  That said, my gut instinct is that since they don’t mention many of the papers (like Geyer’s one) giving methods to do the calculation that this comment is not intended to be the final word on the subject, more so a segue to their presentation of the Bayesian solutions.

Thinking about detection of spectral features, I noticed some bizarre mis-use of the chi square asymptotics by Rivera-Thorsen et al. (2015) who have invented a way to shoe-horn the test on to their fitting when there are only as many data points as the number of parameters in the model: “The minimum number of contributing lines required for our fitting procedure is two. In this case, with two data points and two free parameters, there are zero degrees of freedom and thus the usual reduced χ2 is not defined. Instead, we report a pseudo-reduced χ2 , which is defined as χ2 / ([DOF] + 1).”  Somehow the warnings on abuse of chi squared in astronomy by Andrae et al. (2010) still aren’t getting through.

An interesting aside from my reading in particle physics is that it turns out that when people report a result as “3 sigma significance” (for instance) it doesn’t necessarily mean what I thought it meant.  In particle physics it seems to most often mean the normal quantile of the p-value corresponding to a one sided test (Cowan et al. 2010) whereas I had always thought it meant the p-value corresponding to a two-sided test (as in cases I’m familiar with from astronomy; e.g. King et al. 2012).  When I mentioned this on the astrostatistics facebook group some guy thought I was having a go at particle physics but that really wasn’t the case.  I was surprised: until last week I thought it was one way and not some other way.  If the purpose of converting a p-value to a sigma is to ease interpretation it seems to me that having competing definitions or standard defeats that purpose.  Also, there is not entirely consistency within fields; for a particle physics example of the second type see Aaij et al. (2012).

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s