I noticed this paper on astro ph today by Barclay et al. presenting an model comparison for Kepler-296, examining which of the two stars in this binary system is most likely to host the five detected/inferred planets. What caught my eye was the statement in the abstract that the model probabilities are computed using importance sampling. At face value I supposed this meant that they did MCMC, observed an approximate Gaussianity to the posterior and then just importance sampled from a well-chosen Normal to estimate the normalisation of each model separately. However, it turns out that the problem is structured such that both models share the same parameters and prior support, with the only difference between them being the scale and location of their bivariate Normal priors. In this context the authors’ solution is to run an MCMC chain for a third distribution corresponding to the posterior under flat priors, and then reweight this to estimate the relative marginal likelihood of each of the two models with more specific priors. As seems de rigueur for astronomy this is referred to as ‘importance sampling’ with reference to Hogg et al. (2010).
Personally, I tend call such an algorithm something like ‘pseudo-importance sampling’ or ‘importance sample reweighting’ to acknowledge that the sampling is not iid from the proposal distribution but is rather an approximation via MCMC. The difference being that ordinary importance sampling is unbiased* (and consistent) whereas the MCMC version is only consistent; these being important distinctions for understanding the behaviour of these resulting estimator, especially when used within larger schemes such as my old favourite (the pseudo-marginal method). One version of ordinary importance sampling that the authors could of course have used here would be simply the arithmetic mean estimator formed from the mean likelihood under draws from each of the competing bivariate Normal priors, without invoking the third ‘importance sampling’ prior.
* Assuming one does not normalise the weights.