Approximate Bayesian Computation with heads in the sand

I noticed a recent arXival describing an ABC approach to inferring halo modal parameters from lensing substructure, which it seems is the third in a series of installments, each of which churlishly manages to pretend not to be aware of my seminal ‘ABC for astronomy‘ paper from back in 2012.  While I’m glad to see the ABC technique increasingly being used, consciously, for astronomical analyses, it does seem weird that having figured out that your technique fits within the ABC framework, not to learn from the statistical literature on that topic.  Why I assume no learning has taken place is because the authors in this case haven’t moved beyond simple rejection ABC, which is a massive waste of computational power relative to even a basic population Monte Carlo approach.  Moreover, it seems their scheme is consequently so inefficient that they cannot fit all N of their lenses jointly, instead having to rely on some kind of multiplication of KDE approximations to each subset posterior to form the final posterior.

Heck, if they’d read into the problem from a statistical point of view they might even have learned a bit about how to do subset posteriors better.  For instance, they write that the multiplication of these sub-posteriors is only possible if you use uniform priors, and consequently they go to a lot of trouble to choose transformations allowing for construction of entirely uniform priors.  But, wait for it, what about this from consensus MC: \propto \pi_i(\theta)^{1/N}?!

This entry was posted in Uncategorized. Bookmark the permalink.

3 Responses to Approximate Bayesian Computation with heads in the sand

  1. Tom Loredo says:

    I’m repeatedly bewildered by how little effort many authors put into doing a literature search for prior work relevant to their research (I especially notice this in my role as an assoc. ed. of an applied stats journal that includes papers on statistical applications in astronomy and physics). Firstly, it’s never been *easier* to do a literature search, with tools like Google, ADS, and MathSciNet at hand, not to mention having a world’s worth of colleagues accessible at short notice via email. It’s not like you have to actually go to a library and use card catalogs; what excuse is there for *not* doing a literature search? But more importantly, it’s an issue of basic personal and scientific integrity. If you are writing a paper, presumably that’s because you hope others will read it; courtesy and humility should then suggest it’s your duty to expend at least some effort to read what others have written relevant to your work. But it’s even more basic than that. As Feynman wrote, “The first principle [for producing sound science] is that you must not fool yourself—and you are the easiest person to fool.” He refers to taking extreme care not to fool yourself as a basic “type of integrity” for scientists. Surely the easiest component of this kind of self-skepticism would be doing a literature search, not just on the science but also on your methodology, esp. if it’s nontrivial.

    Turning now to the recent impact of ABC and your seminal 2012 paper: I recently did a hasty bibliographic search to quickly ascertain the impact of ABC in astronomy. I’d been aware of several papers in cosmology and work in progress in exoplanets at a level that was already impressive to me given just 7+ years since the introduction of ABC to astronomy. But my ADS searches turned up several dozen abstracts describing work making substantive use of ABC (I didn’t look at the search results to see how many abstracts were for refereed, published papers; this was just a background info search for a letter of recommendation). That’s a very impressive impact on behalf of a nontrivial statistical approach in astronomy, in just 7+ years .

    I say 7+ years because the first time I heard ABC being advocated for astronomy applications was at the *Statistical Challenges in Modern Astronomy V* meeting in 2011, where Chad Schafer and Peter Freeman gave a solid presentation on it. This was published in the proceedings volume (with discussion by Martin Hendry) in 2012 (ADS:…902….3S/abstract). I was disappointed to see how few citations this paper has in comparison with your 2012 paper. It’s not as ambitious, but still. As far as I can tell, Chad and Peter didn’t post the paper to arXiv. My own experience is that papers I haven’t posted to arXiv are rarely cited. So maybe this result is just saying that arXiv has replaced journals and conference proceedings as primary literature for astronomers. Anyway, this is just to say that the Schafer and Freeman 2011/2012 work probably deserves some credit for recognizing the importance of ABC for astrostatistics and for *trying* to bring it to the attention of astronomers, even if their publication choice minimized its visibility and impact.

    • It’s hard to fathom how much research effort is wasted on duplication of existing work that could have been identified with a quick google search. It’s obviously dissappointing when you come up with an idea for a new method (for example) and then google reveals it’s already been done: but if you search early enough (as you should) then it’s usually straightforward to “pivot” your idea: new application area with a case-specific problem, extension of the method, improvement of efficiency. The worst offending subject area between about 2012 and 2017 was probably machine learning, which was notorious for “re-badging” of existing statistical methods, but my impression is that the two communities now overlap sufficiently that egregious examples will be caught at review.

      Citing Schafer & Freeman was indeed technically difficult for that reason when I was writing my ABC paper; but I’m glad to say we reference their pioneering work in our first footnote!

  2. Pingback: A novel approach to likelihood-free inference: just use the likelihood! | Another Astrostatistics Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s