Comparing the incomparable = polishing the unpolishable* [Guest post: J Michael Burgess]

Trying to produce works that both illuminate the proper use of statistical methodology and perhaps tackle some relevant scientific question is a tough game. Any of us that get comments from the referee such as “why are you using this complicated <insert something from proper stats literature here> when you can just check chi2” knows that it takes an immense amount of momentum to move the field in a statistically sound direction. Yet, the ML buzzword hype is moving into statistics, and astro papers are becoming littered with words such as information criteria and marginalization.  It is great that researchers are aware of these terms.  But like any method in science, they cannot just be thrown around and the words matter. A few improper usages of these words and a trend begins and soon we look back and say “but whats the comparison between the BIC and AIC in your work?” as haphazardly as people ask “what’s your reduced chi2?”. For example, here are a few examples of the common comparison between the AIC and BIC: here, here, and here. The point is that these numbers aren’t telling you the same information (if the BIC is telling you anything!) about the model and data. But this takes a bit deeper reading in the stats literature. But, hey, the same can be said for reduced chi2 or Wilks’ theorem.

Still things can get completely out of hand: here is an example where everything from maximum likelihood, MCMC, distribution, etc are misused so badly that the terms become meaningless. However, with enough of the right stats buzzwords, a reviewer that is not an expert might just accept the sophistication of the work based on the language alone. We have to be careful and investigate both the methods we use, the questions they address and the words associated with them. Otherwise, we might as well use chi-by-eye fits as the random BIC numbers being spewed out are just as meaningless.

*Editor’s note [E.C.]: The silly title is entirely my own; ‘polishing the unpolishable’ is a euphemism for an English expression that describes one’s endeavours when fruitlessly trying to fix something that is fundamentally broken, i.e., ‘polishing a turd’.

This entry was posted in Uncategorized. Bookmark the permalink.

1 Response to Comparing the incomparable = polishing the unpolishable* [Guest post: J Michael Burgess]

  1. Rodrigo Díaz says:

    I couldn’t agree more.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s