The dependence of routine Bayesian model selection methods on irrelevant alternatives

Bayesian methods - either based on Bayes Factors or BIC - are now widely used for model selection. One property that might reasonably be demanded of any model selection method is that if a model M1 is preferred to a model M0, when these two models are expressed as members of one model class M, this preference is preserved when they are embedded in a different class M'. However, we illustrate in this paper that with the usual implementation of these common Bayesian procedures this property does not hold true even approximately. We therefore contend that to use these methods it is first necessary for there to exist a "natural" embedding class. We argue that in any context like the one illustrated in our running example of Bayesian model selection of binary phylogenetic trees there is no such embedding.