On the use of receiver operating character tests for evaluating spatial earthquake forecasts

Spatial forecasts of triggered earthquake distributions have been ranked using receiver operating characteristic (ROC) tests. The test is a binary comparison between regions of positive and negative forecast against positive and negative presence of earthquakes. Forecasts predicting only positive changes score higher than Coulomb methods, which predict positive and negative changes. I hypothesize that removing the possibility of failures in negative forecast realms yields better ROC scores. I create a ‘perfect’ Coulomb forecast where all earthquakes only fall into positive stress change areas and compare with an informationless all-positive forecast. The ‘perfect’ Coulomb forecast barely beats the informationless forecast, and adding as few as 4 earthquakes occurring in the negative stress regions causes the Coulomb forecast to be no better than an informationless forecast under a ROC test. ROC tests also suffer from data imbalance when applied to earthquake forecasts because there are many more negative cases than positive.