Elsevier

Ecological Modelling

Volume 186, Issue 3, 25 August 2005, Pages 280-289
Ecological Modelling

The evaluation strip: A new and robust method for plotting predicted responses from species distribution models

https://doi.org/10.1016/j.ecolmodel.2004.12.007Get rights and content

Abstract

Increasing use is being made in conservation management of statistical models that couple extensive collections of species and environmental data to make predictions of the geographic distributions of species. While the relationships fitted between a species and its environment are relatively transparent for many of these modeling techniques, others are more ‘black box’ in character, only producing geographic predictions and providing minimal or untraditional summaries of the fitted relationships on which these predictions are based. This in turn prevents robust evaluation of the ecological sensibility of such models, a necessary process if model predictions are to be treated with confidence. Here we propose a new but simple method for visualizing modeled responses that can be implemented with any modeling method, and demonstrate its application using five common methods applied to the prediction of an Australian tree species. This is achieved by insetting an “evaluation strip” into the spatial data layers, which, after predictions have been made, can be clipped out and used for creating plots of the modelled responses. We present findings of the application strip for algorithms GLMs, GAMs, CLIM, DOMAIN and MARS. Evaluation strips can be constructed to investigate either uni-variate responses, or the simultaneous variation in predicted values in relation to two variables. The latter option is particularly useful for evaluating responses in models that allow the fitting of complex interaction terms.

Introduction

Statistical models are frequently used to relate species’ presence, presence–absence or abundance, to the environment at some set of survey sites, and these models can then be used to predict the geographic distribution of the species over some region, given sufficient data over the area of interest (Guisan and Zimmerman, 2000). A wide variety of methods are used to develop such models and their predictions (Elith and Burgman, 2003, Rushton et al., 2004), including climate envelopes (Busby, 1991, Kadmon et al., 2003), multivariate association methods (Carpenter et al., 1993, Hirzel, 2001, ter Braak, 1986), regression models (Hastie, 1991, McCullagh and Nelder, 1989), tree-based models (Breiman et al., 1984), machine-learning methods (Aleksander and Morton, 1990, Mitchell, 1996), and ensemble methods such as model averaging (Burnham and Anderson, 2002), bagging (Breiman, 1996) and boosting (Schapire, 1990).

Some modelling methods are more transparent or interpretable than others. For example, relationships fitted by most regression techniques can be graphed and assessed for ecological rationality, making the modelled relationships transparent and open to scrutiny (e.g., Austin, 2002). In contrast, other methods (such as rule sets derived from genetic algorithms, and neural networks) are often called “black boxes” – model details are concealed from the modeller under normal operation, making them both more difficult to interpret and assess, and prone to over-fitting and/or over-explanation (Venables and Ripley, 1999, p. 302). However, these methods may have other desirable characteristics relating, for example, to software availability, speed of computation, or the ability to handle complex interactions between predictors.

Where predictions from such models are used in conservation planning and land management (e.g., Araujo and Williams, 2000, Buckland et al., 1996, Venier et al., 1998) it is usually important that users have an understanding of the structure and predictive accuracy of models, so that they can be realistic about what can be achieved and can allow for uncertainty. In most cases evaluations of predictive performance focus on a comparison of predictions against observations at a particular set of sites (Fielding and Bell, 1997). Although such evaluation data should ideally be independent, or formed through resampling the modelling data, in many cases models are evaluated using the training data itself. A broad range of statistics such as kappa, area under the receiver operating characteristic (ROC) curve and correlation coefficients can be used to assess whether predictions are suitably accurate for their intended use (Moisen and Frescino, 2002, Pearce and Ferrier, 2000, Rushton et al., 2004). However, these types of evaluations are somewhat restricted because evaluation data are always limited in number, and, because they do not assess the predictions in geographic space, do not allow for exploration of spatial errors (Elith et al., 2002, Fielding and Bell, 1997) – nor do they comprehensively sample environmental space. The method we propose here addresses this by facilitating the visualisation of predicted responses of species to environmental variables, even where these relationships are not readily apparent from the model itself.

Others have demonstrated the usefulness of plotting modelled responses for generalized linear models (GLMs) and generalized additive models (GAMs). These plots are simply a visualisation of the model that relates the response (e.g., species presence or abundance) to a set of variables. If a model related abundance of a species to, say, soil fertility and aspect, then there would be two plotted responses: one between abundance and fertility, the other between abundance and aspect. Austin et al. (1995) called them ‘prediction curves’ or ‘response curves’, constructed them for GLMs and GAMs developed on simulated abundance data, and used them to assess predicted responses compared with the known responses. Ferrier et al., 2000, Ferrier et al., 2002 used GAMs to model presence–absence data, and included graphs of the response to single variables (with confidence intervals) as one of a suite of methods for evaluating models and understanding uncertainties in them. These authors developed the plotted responses in similar ways – they held all variables other than the variable of interest constant (at its mean or some other appropriate value), and made predictions at regular intervals over the range of the remaining variable. The methods for plotting the component terms of GLMs and GAMs in software such as S-PLUS (Mathsoft, 1999) are based on the same idea. There are also examples where other methods have been used to model the response to one or two variables, and a prediction surface has been plotted (Elder, 2003).

Here we generalize this approach by proposing a technique that allows fitted responses to be made transparent, even when using methods that only predict to gridded (raster) data – for example, DOMAIN (Carpenter et al., 1993) and the genetic algorithm GARP (Stockwell and Peters, 1999). Currently, if one uses such methods for modelling, the only predictions that can be retrieved are the habitat maps that are produced. Visualising the responses fitted between species occurrence/abundance and environment from these models is not simple (we found no examples in the literature), and this can severely limit how well they can be assessed for ecological realism (Austin, 2002) or compared with other more transparent models. Our proposed protocol complements other model evaluation methods that focus more on the fit of predictions to existing or new data.

Section snippets

Materials and methods

Plotted responses are generated here for a case study in north-east New South Wales, Australia, for the tree species, Eucalyptus carnea. Both the region, and the environmental and species data we use have been previously described by Ferrier et al. (2002). In the current application, we used 13 environmental variables (predictors) for modelling, and the species data consisted of 86 presence records. Environmental data consisted of raster grids (about 5000 by 3500, 100 m cells) stored in a GIS.

Results

The modelled responses to the variables annual mean temperature and ruggedness are presented for each of the five modelling methods, using plot type 1: response to one variable (Fig. 2). Results for the first column – the response to temperature – clearly show variation in the shape of the modelled responses between the different techniques. The CLIM prediction (from the climate envelope model, Table 1) can be used as a guide to the frequency of the species along the environmental gradient: a

Discussion

This approach is a simple but useful extension to existing modelling techniques. These plots could already be made for many current modelling methods with existing abilities to predict to tabular data. However, there are a growing number of modelling techniques that only predict to raster data, and these are gaining popularity – assessing the ecological validity of the relationships fitted by such models is currently frustrated by our difficulty in visualising the ecological responses they fit.

Conclusion

While assessment of the ecological validity of species:environment models is straight-forward for many of the traditional (regression-based) modelling methods, it is much more difficult for recently developed modelling tools that predict directly to gridded spatial data, and produce minimal summaries of fitted relationships. Our proposed evaluation strip overcomes this difficulty by inserting data strips into spatial grids so plots can be made for any method. This will in turn aid model

Acknowledgements

This work was developed within the working group on modelling species distributions led by Town Peterson and Craig Moritz at the National Center for Ecological Analysis and Synthesis. The data were provided by SF, from the New South Wales Department of Environment and Conservation. GLMs and GAMs were modelled by Anthony Lehmann, DOMAIN by Bette Loiselle, Chrissy Howell and JE, CLIM by Catherine Graham, and MARS by JE, JL and FH. Mary Wisz contributed to data organisation. Stuart Elith prepared

References (49)

  • M.P. Austin et al.

    Modelling of Landscape Patterns and Processes Using Biological Data. Subproject 5: Simulated Data Case Study

    (1995)
  • L. Breiman

    Bagging predictors

    Mach. Learn.

    (1996)
  • L. Breiman et al.

    Classification and Regression Trees

    (1984)
  • S.T. Buckland et al.

    Predicting distributional change, with application to bird distributions on northeast Scotland

    Global Ecol. Biogeogr.

    (1996)
  • K.P. Burnham et al.

    Model Selection and Inference: A Practical Information-Theoretic Approach

    (2002)
  • J.R. Busby

    BIOCLIM – a bioclimate analysis and prediction system

  • G. Carpenter et al.

    DOMAIN: a flexible modelling procedure for mapping potential distributions of plants and animals

    Biodivers. Conserv.

    (1993)
  • A.C. Davison et al.

    Bootstrap Methods and their Application

    (1997)
  • J.F. Elder

    The generalization paradox of ensembles

    J. Comput. Graphical Stat.

    (2003)
  • J. Elith et al.

    Predictions and their validation: rare plants in the Central Highlands, Victoria, Australia

  • J. Elith et al.

    Habitat models for PVA

  • M. Ennis et al.

    A comparison of statistical learning methods on the GUSTO database

    Stat. Med.

    (1998)
  • Ferrier, S., Watson, G., 1996. An Evaluation of the Effectiveness of Environmental Surrogates and Modelling Techniques...
  • Ferrier, S., Watson, G., Hines, H., Brown, D., 2000. Predictive modelling of biological data. In: Brown, D., Hines, H.,...
  • Cited by (197)

    • Habitat suitability mapping of the Indian giant flying squirrel (Petaurista philippensis Elliot, 1839) in India with ensemble modeling

      2023, Acta Ecologica Sinica
      Citation Excerpt :

      Following that, we calculated these categories area at the country scale, in biogeographic zones, and state wise to assess the availability of suitable area for IGFS. The response curves for each variable used in the modeling were also prepared at country level using the evaluation strip method in the r-environment with the Biomod2 package to see the trend and directionality [40]. TSS score for all 50 replicated models ranged from 0.714 to 0.813, and ROC score ranged from 0.892 to 0.945, indicating good performance (Fig. 2).

    View all citing articles on Scopus
    View full text