The evaluation strip: A new and robust method for plotting predicted responses from species distribution models
Introduction
Statistical models are frequently used to relate species’ presence, presence–absence or abundance, to the environment at some set of survey sites, and these models can then be used to predict the geographic distribution of the species over some region, given sufficient data over the area of interest (Guisan and Zimmerman, 2000). A wide variety of methods are used to develop such models and their predictions (Elith and Burgman, 2003, Rushton et al., 2004), including climate envelopes (Busby, 1991, Kadmon et al., 2003), multivariate association methods (Carpenter et al., 1993, Hirzel, 2001, ter Braak, 1986), regression models (Hastie, 1991, McCullagh and Nelder, 1989), tree-based models (Breiman et al., 1984), machine-learning methods (Aleksander and Morton, 1990, Mitchell, 1996), and ensemble methods such as model averaging (Burnham and Anderson, 2002), bagging (Breiman, 1996) and boosting (Schapire, 1990).
Some modelling methods are more transparent or interpretable than others. For example, relationships fitted by most regression techniques can be graphed and assessed for ecological rationality, making the modelled relationships transparent and open to scrutiny (e.g., Austin, 2002). In contrast, other methods (such as rule sets derived from genetic algorithms, and neural networks) are often called “black boxes” – model details are concealed from the modeller under normal operation, making them both more difficult to interpret and assess, and prone to over-fitting and/or over-explanation (Venables and Ripley, 1999, p. 302). However, these methods may have other desirable characteristics relating, for example, to software availability, speed of computation, or the ability to handle complex interactions between predictors.
Where predictions from such models are used in conservation planning and land management (e.g., Araujo and Williams, 2000, Buckland et al., 1996, Venier et al., 1998) it is usually important that users have an understanding of the structure and predictive accuracy of models, so that they can be realistic about what can be achieved and can allow for uncertainty. In most cases evaluations of predictive performance focus on a comparison of predictions against observations at a particular set of sites (Fielding and Bell, 1997). Although such evaluation data should ideally be independent, or formed through resampling the modelling data, in many cases models are evaluated using the training data itself. A broad range of statistics such as kappa, area under the receiver operating characteristic (ROC) curve and correlation coefficients can be used to assess whether predictions are suitably accurate for their intended use (Moisen and Frescino, 2002, Pearce and Ferrier, 2000, Rushton et al., 2004). However, these types of evaluations are somewhat restricted because evaluation data are always limited in number, and, because they do not assess the predictions in geographic space, do not allow for exploration of spatial errors (Elith et al., 2002, Fielding and Bell, 1997) – nor do they comprehensively sample environmental space. The method we propose here addresses this by facilitating the visualisation of predicted responses of species to environmental variables, even where these relationships are not readily apparent from the model itself.
Others have demonstrated the usefulness of plotting modelled responses for generalized linear models (GLMs) and generalized additive models (GAMs). These plots are simply a visualisation of the model that relates the response (e.g., species presence or abundance) to a set of variables. If a model related abundance of a species to, say, soil fertility and aspect, then there would be two plotted responses: one between abundance and fertility, the other between abundance and aspect. Austin et al. (1995) called them ‘prediction curves’ or ‘response curves’, constructed them for GLMs and GAMs developed on simulated abundance data, and used them to assess predicted responses compared with the known responses. Ferrier et al., 2000, Ferrier et al., 2002 used GAMs to model presence–absence data, and included graphs of the response to single variables (with confidence intervals) as one of a suite of methods for evaluating models and understanding uncertainties in them. These authors developed the plotted responses in similar ways – they held all variables other than the variable of interest constant (at its mean or some other appropriate value), and made predictions at regular intervals over the range of the remaining variable. The methods for plotting the component terms of GLMs and GAMs in software such as S-PLUS (Mathsoft, 1999) are based on the same idea. There are also examples where other methods have been used to model the response to one or two variables, and a prediction surface has been plotted (Elder, 2003).
Here we generalize this approach by proposing a technique that allows fitted responses to be made transparent, even when using methods that only predict to gridded (raster) data – for example, DOMAIN (Carpenter et al., 1993) and the genetic algorithm GARP (Stockwell and Peters, 1999). Currently, if one uses such methods for modelling, the only predictions that can be retrieved are the habitat maps that are produced. Visualising the responses fitted between species occurrence/abundance and environment from these models is not simple (we found no examples in the literature), and this can severely limit how well they can be assessed for ecological realism (Austin, 2002) or compared with other more transparent models. Our proposed protocol complements other model evaluation methods that focus more on the fit of predictions to existing or new data.
Section snippets
Materials and methods
Plotted responses are generated here for a case study in north-east New South Wales, Australia, for the tree species, Eucalyptus carnea. Both the region, and the environmental and species data we use have been previously described by Ferrier et al. (2002). In the current application, we used 13 environmental variables (predictors) for modelling, and the species data consisted of 86 presence records. Environmental data consisted of raster grids (about 5000 by 3500, 100 m cells) stored in a GIS.
Results
The modelled responses to the variables annual mean temperature and ruggedness are presented for each of the five modelling methods, using plot type 1: response to one variable (Fig. 2). Results for the first column – the response to temperature – clearly show variation in the shape of the modelled responses between the different techniques. The CLIM prediction (from the climate envelope model, Table 1) can be used as a guide to the frequency of the species along the environmental gradient: a
Discussion
This approach is a simple but useful extension to existing modelling techniques. These plots could already be made for many current modelling methods with existing abilities to predict to tabular data. However, there are a growing number of modelling techniques that only predict to raster data, and these are gaining popularity – assessing the ecological validity of the relationships fitted by such models is currently frustrated by our difficulty in visualising the ecological responses they fit.
Conclusion
While assessment of the ecological validity of species:environment models is straight-forward for many of the traditional (regression-based) modelling methods, it is much more difficult for recently developed modelling tools that predict directly to gridded spatial data, and produce minimal summaries of fitted relationships. Our proposed evaluation strip overcomes this difficulty by inserting data strips into spatial grids so plots can be made for any method. This will in turn aid model
Acknowledgements
This work was developed within the working group on modelling species distributions led by Town Peterson and Craig Moritz at the National Center for Ecological Analysis and Synthesis. The data were provided by SF, from the New South Wales Department of Environment and Conservation. GLMs and GAMs were modelled by Anthony Lehmann, DOMAIN by Bette Loiselle, Chrissy Howell and JE, CLIM by Catherine Graham, and MARS by JE, JL and FH. Mary Wisz contributed to data organisation. Stuart Elith prepared
References (49)
- et al.
Selecting areas for species persistence using occurrence data
Biol. Conserv.
(2000) Spatial prediction of species distribution: an interface between ecological theory and statistical modelling
Ecol. Model.
(2002)- et al.
Mapping epistemic uncertainties and vague concepts in predictions of species distribution
Ecol. Model.
(2002) - et al.
Predictive habitat distribution models in ecology
Ecol. Model.
(2000) - et al.
GRASP: generalized regression analysis and spatial prediction
Ecol. Model.
(2002) - et al.
Comparing five modeling techniques for predicting forest characteristics
Ecol. Model.
(2002) - et al.
Evaluating the predictive performance of habitat models developed using logistic regression
Ecol. Model.
(2000) - et al.
Modelling abundance and distribution of Marbled Murrelets (Brachyramphus marmoratus) using GIS, marine data and advanced multivariate statistics
Ecol. Model.
(2004) - et al.
Predicting species distribution using presence-only data: a case study of native New Zealand ferns
Ecol. Model.
(2002) - et al.
An Introduction to Neural Computing
(1990)
Modelling of Landscape Patterns and Processes Using Biological Data. Subproject 5: Simulated Data Case Study
Bagging predictors
Mach. Learn.
Classification and Regression Trees
Predicting distributional change, with application to bird distributions on northeast Scotland
Global Ecol. Biogeogr.
Model Selection and Inference: A Practical Information-Theoretic Approach
BIOCLIM – a bioclimate analysis and prediction system
DOMAIN: a flexible modelling procedure for mapping potential distributions of plants and animals
Biodivers. Conserv.
Bootstrap Methods and their Application
The generalization paradox of ensembles
J. Comput. Graphical Stat.
Predictions and their validation: rare plants in the Central Highlands, Victoria, Australia
Habitat models for PVA
A comparison of statistical learning methods on the GUSTO database
Stat. Med.
Cited by (197)
Temporal dynamics in gray wolf space use suggest stabilizing range in the Great Lakes region, USA
2024, Biological ConservationHigh-resolution assessment of retrogressive thaw slump susceptibility in the Qinghai-Tibet Engineering Corridor
2023, Research in Cold and Arid RegionsNorthern spotted owl nesting habitat under high potential wildfire threats along the California Coastal Redwood Forest
2023, Science of the Total EnvironmentHabitat suitability mapping of the Indian giant flying squirrel (Petaurista philippensis Elliot, 1839) in India with ensemble modeling
2023, Acta Ecologica SinicaCitation Excerpt :Following that, we calculated these categories area at the country scale, in biogeographic zones, and state wise to assess the availability of suitable area for IGFS. The response curves for each variable used in the modeling were also prepared at country level using the evaluation strip method in the r-environment with the Biomod2 package to see the trend and directionality [40]. TSS score for all 50 replicated models ranged from 0.714 to 0.813, and ROC score ranged from 0.892 to 0.945, indicating good performance (Fig. 2).
Spatial distributions, environmental drivers and co-existence patterns of key cold-water corals in the deep sea of the Azores (NE Atlantic)
2023, Deep-Sea Research Part I: Oceanographic Research Papers