Is your ad hoc model selection strategy affecting your multimodel inference?

(Yackulic) 1. Ecologists routinely fit complex models with multiple parameters of interest, where hundreds or more competing models are plausible. To limit the number of fitted models, ecologists often define a model selection strategy composed of a series of stages in which certain features of a model are compared while other features are held constant. Defining these multi-stage strategies requires making a series of decisions, which may potentially impact inferences, but have not been critically evaluated.
2. We begin by identifying key features of strategies, introducing descriptive terms when they did not already exist in the literature. Strategies differ in how they define and order model building stages. Sequential-by-sub-model strategies focus on one sub-model (parameter) at a time with modeling of subsequent sub-models dependent on the selected model structures from the previous stages. Secondary candidate set strategies model sub-models independently and combine the top set of models from each sub-model for selection in a final stage. Build-up approaches define stages across sub-models and increase in complexity at each stage. Strategies also differ in how the top set of models is selected in each stage and whether they use null or more complex model structures for non-target sub-models.
3. We tested the performance of different model selection strategies using four datasets and three model types. For each dataset, we determined the “true” distribution of AIC weights by fitting all plausible models. Then, we calculated the number of models that would have been fitted and the portion of “true” AIC weight we recovered under different model selection strategies.
4. Sequential-by-sub-model strategies often performed poorly. Build-up or secondary candidate sets were more reliable, provided all models within 5 AIC of the top model were carried forward to subsequent stages. The structure of non-target sub-models was less important.
5. Multi-stage approaches cannot compensate for a lack of critical thought in selecting covariates and building models to represent competing a priori hypotheses. However, even when competing hypotheses for different sub-models are limited, thousands or more models may be possible so strategies to explore candidate model space reliably and efficiently will be necessary.