NAME: Cohen’s Kappa and Classification Table Metrics 2.1a:
           An ArcView 3x Extension for Accuracy Assessment of Spatially-Explicit Models 
          (Click name to download)

Aka: kappa_stats.avx

Last modified: December 10, 2007

TOPICS: Kappa, classification, accuracy, sensitivity, specificity, omission, commission, user accuracy, producer accuracy, P-value, sample size, model, statistics, distributions, t, F, logistic, normal, skewness, kurtosis, binomial, probability, critical, Poisson, chi-square

AUTHORS:

DESCRIPTION:  

In the field of spatially explicit modeling, well-developed accuracy assessment methodologies are often poorly applied. Deriving model accuracy metrics have been possible for decades, but these calculations were made by hand or with the use of a spreadsheet application. Accuracy assessments may be useful for: (1) ascertaining the quality of a model; (2) improving model quality by identifying and correcting sources of error; (3) facilitating a comparison of various algorithms, techniques, model developers and interpreters; and, (4) determining the utility of the data product in a decision-making context. When decisions are made with models of unknown or poorly-assessed accuracy, resource managers run the risk of making wrong decisions or drawing erroneous conclusions. Untested predictive surface maps should be viewed as untested hypotheses and, by extension, poorly tested predictive models are poorly tested hypotheses. Often, if any accuracy measure is provided at all, only the overall model accuracy is reported. However, numerous accuracy metrics are available which can describe model accuracy and performance. Because issues concerning data quality and model accuracy in landscape analyses have received little attention in the management literature, we found it useful to develop a systematic and robust procedure for assessing the accuracy of spatially explicit models. We created an ArcView 3.x extension that provides end users with a packaged approach for accuracy assessment, using Cohen’s Kappa statistic as well as several other metrics including overall accuracy, overall misclassification rate, model specificity and sensitivity, omission and commission errors, and positive and negative predictive power. Collectively, these metrics may be used for gauging model performance. When multiple models are available, these metrics offer end users the ability to quantitatively compare and identify the “best” model within a multi-criteria model selection process.

This extension offers several functions to aid in accuracy assessment, as well as several general statistical tools.  Please review the manual for a full discussion of all functions, but in summary they are:

Acknowledgments: The tools in this extension are based primarily on functions described in “Assessing the Accuracy of Remotely Sensed Data: Principles and Practices” by Russell G. Congalton and Kass Green (Congalton and Green 1999).

Special thanks to Dr. William Block, USDA FS, Rocky Mountain Research Station for use of the retrospective dataset used in the case study. Wynne (2003) used these data as the basis of WHR model development for his M.S. thesis research. Also, special thanks to Mr. Rudy King and Ms. Kristen Covert-Bratland, also of the Rocky Mountain Research Station, for assistance in statistical questions.

Certain tools (esp. the Field Statistics and the histogram) were originally developed by the author for the University of Arizona’s Saguaro project (see http://saguaro.geo.arizona.edu/) and are included with their permission. The authors thank Mr. Larry Kendall of the University of Arizona, and Mr. Scott Walker of Northern Arizona University, for their help in developing those tools and their willingness to share them.

Special thanks also to Dr. John Prather and Dr. Russell Congalton for reviewing early drafts of this manual, and to Dr. Congalton for suggesting the correction for locational uncertainty.

The Statistical Probability tools are almost identical to those in the author’s Statistics and Probability Tools extension (see http://www.jennessent.com/arcview/stats_dist.htm) and are included because they enhance and complement the statistical functions. The manual for that extension has also been incorporated into this manual.

REQUIRES: ArcView 3.x; Spatial Analyst

This extension also requires the file "avdlog.dll" be present in the ArcView/BIN32 directory (or $AVBIN/avdlog.dll) and the Dialog Designer extension be located in your ArcView/ext32 directory, which they usually are if you're running AV 3.1 or better. The Dialog Designer doesn't have to be loaded; it just has to be available. If you are running AV 3.0a, you can download the appropriate files for free from ESRI at:

http://support.esri.com/index.cfm?fa=downloads.patchesServicePacks.viewPatch&PID=25&MetaID=483

REVISIONS:  Version 2.1 (May 23, 2006) includes minor code revisions to avoid problems with Chinese versions of Windows, as well as adds menu items to search the Kappa scripts for errors.

Version 2.1a (December 10, 2007): Minor update to allow extension to work with PointZ, PointM, PolygonZ and PolyonM shapefiles.

Recommended Citation Format:  For those who wish to cite this extension, the authors recommend something similar to:

Jenness, J. and J. J. Wynne. 2006. Kappa analysis (kappa_stats.avx) extension for ArcView 3.x. Jenness Enterprises. Available at: http://www.jennessent.com/arcview/kappa_stats.htm.

Please let us know if you cite this extension in a publication (jeffj@jennessent.com or jwynne@usgs.gov), so we may update the citation list accordingly.

Enjoy! Please contact the authors if you have problems or find bugs.