Another model selection paper on astro-ph today, this one by Veneziani et al. & concerning far-infrared SED fitting. According to the authors’ own comments this is the third iteration of the paper through the refereeing process & (IMHO) no wonder, the semi-Bayesian model selection procedure used here is not very attractive. In particular, the authors run an initial MCMC exploration of the parameter space for each of two competing models given non-informative priors, select the “best-fitting” one based on a “chi-squared” (statisticians read here “sum of squared standard errors”) comparison at the posterior mode, then proceed to refit the chosen model using priors based on the initial posterior. The obvious problems being the mish-mash of Bayesian inference with frequentist model selection (and indeed with a frequentist technique that is best avoided; e.g. Andrae et al.), and the using-the-data-twice update of the “prior”.
However, it was yet another aspect of the analysis which perhaps troubled me most: the authors’ treatment of calibration errors. In particular, Veneziani et al. propose to allow for the potential impact of systematic calibration errors in each of their eight spectral bands by including an eight parameter calibration component into the likelihood model, assigning a prior for these eight new parameters, and integrating them out to recover the marginal posterior for the original parameters of interest. At face value, a perfectly justified, Bayesian procedure. But, let’s look closely at the details. First, the priors. While uniform priors are often favoured by experimentalists as being ‘relatively’ non-informative and thus seemingly more ‘objective’, they are rarely well-motivated from a physical perspective. Here the choice of independent (fair enough) uniform priors on each calibration offset seems most unusual; is it really fair to say that a priori we would assign equal relative probability to all possible calibration offsets in the set range? and why is this range are all equally plausible but an epsilon step outside this range is completely implausible?
Second, there is the MC integration over this eight dimensional parameter space. The authors’ attempt this by drawing 100 8-tuples from this (of course) eight dimensional parameter space and note that little change to the resulting estimator is achieved by using more than 100 draws. So … either the calibration errors play such a minimal role in the likelihood function that 100 draws is sufficient to take in the bulk of posterior mass in the marginalised calibration space, or in fact the opposite is true and that the bulk of mass lies in such a small region of the calibration space that whether you try 100 or 1000 random draws from the uniform prior you’re never likely to notice it! Hmmm ….
On the plus side, the “astro-statistics” community was well rewarded with citations; namely, for MCMC (Lewis & Bridle) and hierarchical models (Kelly et al.), because, like, we invented these techniques … right? Well, at least I’m still getting citations for binomial population proportions!