14.1.5.1 Error Estimation, Classification Accuracy

Pearl, J.[Judea],
Capacity and Error Estimates for Boolean Classifiers with Limited Complexity,
PAMI(1), No. 4, October 1979, 350-356. BibRef 7910

Pearl, J.[Judea],
An application of rate-distortion theory to pattern recognition and classification,
PR(8), No. 1, January 1976, pp. 11-22.
WWW Version. 0309
BibRef

McLachlan, G.J.,
A note on the choice of a weighting function to give an efficient method for estimating the probability of misclassification,
PR(9), No. 3, October 1977, pp. 147-149.
WWW Version. 0309
BibRef

Bock, H.H.,
On some significance tests in cluster analysis,
Classification(2), 1985, pp. 77-108.
Springer DOI Link BibRef 8500

Lawoko, C.R.O., McLachlan, G.J.,
Asymptotic error rates of the W and Z statistics when the training observations are dependent,
PR(19), No. 6, 1986, pp. 467-471.
WWW Version. 0309
BibRef

Ganesalingam, S., McLachlan, G.J.,
Error rate estimation on the basis of posterior probabilities,
PR(12), No. 6, 1980, pp. 405-413.
WWW Version. 0309
BibRef

Chittineni, C.B.,
On the estimation of probability of error,
PR(9), No. 4, 1977, pp. 191-196.
WWW Version. 0309
BibRef

Chittineni, C.B.,
Estimation of probabilities of label imperfections and correction of mislabels,
PR(13), No. 3, 1981, pp. 257-268.
WWW Version. 0309
BibRef

van Otterloo, P.J., Young, I.T.,
A distribution-free geometric upper bound for the probability of error of a minimum distance classifier,
PR(10), No. 4, 1978, pp. 281-286.
WWW Version. 0309
BibRef

Glick, N.[Ned],
Additive estimators for probabilities of correct classification,
PR(10), No. 3, 1978, pp. 211-222.
WWW Version. 0309
BibRef

Engvall, J.L.[John L.],
A least upper bound for the average classification accuracy of multiple observers,
PR(12), No. 6, 1980, pp. 415-419.
WWW Version. 0309
BibRef

Kittler, J.V., Devijver, P.A.,
An efficient estimator of pattern recognition system error probability,
PR(13), No. 3, 1981, pp. 245-249.
WWW Version. 0309
BibRef

Lahart, M.J.,
Estimation of Error Rates in Classification of Distorted Imagery,
PAMI(6), No. 4, July 1984, pp. 535-542. BibRef 8407

Fukunaga, K., and Flick, T.E.,
Classification Error for a Very Large Number of Classes,
PAMI(6), No. 6, November 1984, pp. 779-788. See also Optimal Global Nearest Neighbor Metric, An. BibRef 8411

Fukunaga, K., and Hayes, R.R.,
Estimation of Classifier Performance,
PAMI(11), No. 10, October 1989, pp. 1087-1101.
IEEE Abstract. IEEE Top Reference.
WWW Version. BibRef 8910

Pawlak, M.[Miroslaw],
On the asymptotic properties of smoothed estimators of the classification error rate,
PR(21), No. 5, 1988, pp. 515-524.
WWW Version. 0309
BibRef

Pawlak, M., Liao, X.,
Estimation of error rates using smoothed estimators,
ICPR88(II: 954-956).
IEEE DOI Link 8811
BibRef

Devroye, L.,
Automatic Pattern Recognition: A Study of the Probability of Error,
PAMI(10), No. 4, July 1988, pp. 530-543.
IEEE Abstract. IEEE Top Reference.
WWW Version. BibRef 8807

Devroye, L., Gyorfi, L., Lugosi, G.,
Probabilistic Theory of Pattern Recognition,
Springer-Verlag1996. BibRef 9600

Zhu, Q.M.[Qiu-Ming],
On the minimum probability of error of classification with incomplete patterns,
PR(23), No. 11, 1990, pp. 1281-1290.
WWW Version. 0401
BibRef

Kalkanis, G., Conroy, G.V.,
Interval Error Estimators in Class Probability Trees,
PRL(17), No. 7, June 10 1996, pp. 705-712. 9607
BibRef

Durso, G., Menenti, M.,
Performance Indicators for the Statistical Evaluation of Digital Image Classifications,
PandRS(51), No. 2, April 1996, pp. 78-90. 9605
BibRef

Pal, N.R., Biswas, J.,
Cluster Validation Using Graph-Theoretic Concepts,
PR(30), No. 6, June 1997, pp. 847-857.
WWW Version. 9706
BibRef

Kloditz, C., Vanboxtel, A., Carfagna, E., Vandeursen, W.,
Estimating the Accuracy of Coarse Scale Classification Using High Scale Information,
PhEngRS(64), No. 2, February 1998, pp. 127-133. 9803
BibRef

Bax, E.,
Validation of Average Error Rate over Classifiers,
PRL(19), No. 2, February 1998, pp. 127-132. 9808
BibRef

Bax, E.[Eric],
Improved Hoeffding-style performance guarantees for accurate classifiers,
PRL(20), No. 4, April 1999, pp. 445-449. BibRef 9904

Bouchaffra, D.[Djamel], Govindaraju, V.[Venu], Srihari, S.[Sargur],
A Methodology for Mapping Scores to Probabilities,
PAMI(21), No. 9, September 1999, pp. 923-927.
IEEE Abstract. IEEE Top Reference.
WWW Version. BibRef 9909
Earlier:
A Methodology for Deriving Probabilistic Correctness Measures from Recognizers,
CVPR98(930-935).
IEEE Abstract. IEEE Top Reference. Derive a probability of correctness that can be compared across all classifiers. BibRef

Tulyakov, S., Govindaraju, V.,
Combining matching scores in identification model,
ICDAR05(II: 1151-1155).
IEEE DOI Link 0508
Combining scores. Best score not always best, depending on number of options. BibRef

Ho, T.K., Basu, M.,
Complexity Measures of Supervised Classification Problems,
PAMI(24), No. 3, March 2002, pp. 289-300.
IEEE Abstract. IEEE Top Reference.
WWW Version. 0202
BibRef
Earlier:
Measuring the Complexity of Classification Problems,
ICPR00(Vol II: 43-47).
IEEE DOI Link
HTML Version. 0009
BibRef

Ho, T.K.[Tin Kam],
Data Complexity Analysis: Linkage between Context and Solution in Classification,
SSPR08(986-995).
Springer DOI Link 0812
BibRef
And: SSPR08(1).
Springer DOI Link 0812
BibRef

Clarkson, E.[Eric],
Bounds on the area under the receiver operating characteristic curve for the ideal observer,
JOSA-A(19), No. 10, October 2002, pp. 1963-1968.
WWW Version. 0210
BibRef

Clarkson, E.[Eric],
Estimation receiver operating characteristic curve and ideal observers for combined detection/estimation tasks,
JOSA-A(24), No. 12, December 2007, pp. B91-B98.
WWW Version. 0801
BibRef

Berikov, V.B.[Vladimir B.], Litvinenko, A.[Alexander],
The influence of prior knowledge on the expected performance of a classifier,
PRL(24), No. 15, November 2003, pp. 2537-2548.
WWW Version. 0308
See also approach to the evaluation of the performance of a discrete classifier, An. BibRef

Dougherty, E.R.[Edward R.], Brun, M.[Marcel],
A probabilistic theory of clustering,
PR(37), No. 5, May 2004, pp. 917-925.
WWW Version. 0405
BibRef

Braga-Neto, U.M.[Ulisses M.], Dougherty, E.R.[Edward R.],
Bolstered error estimation,
PR(37), No. 6, June 2004, pp. 1267-1281.
WWW Version. 0405
For further info:
WWW Version. BibRef

Braga-Neto, U.M.[Ulisses M.], Dougherty, E.R.[Edward R.],
Exact performance of error estimators for discrete classifiers,
PR(38), No. 11, November 2005, pp. 1799-1814.
WWW Version. 0509
BibRef

Zollanvari, A.[Amin], Braga-Neto, U.M.[Ulisses M.], Dougherty, E.R.[Edward R.],
On the sampling distribution of resubstitution and leave-one-out error estimators for linear classifiers,
PR(42), No. 11, November 2009, pp. 2705-2723.
Elsevier DOI Link
WWW Version. 0907
Error estimation; Parametric classification; Linear discriminant analysis; Sampling distribution; Resubstitution; Leave-one-out BibRef

Brun, M.[Marcel], Sima, C.[Chao], Hua, J.P.[Jian-Ping], Lowey, J.[James], Carroll, B.[Brent], Suh, E.[Edward], Dougherty, E.R.[Edward R.],
Model-based evaluation of clustering validation measures,
PR(40), No. 3, March 2007, pp. 807-824.
WWW Version. 0611
Clustering algorithms; Clustering errors; Validation indices BibRef

Edwards, D.C., Metz, C.E., Kupinski, M.A.,
Ideal Observers and Optimal ROC Hypersurfaces in N-Class Classification,
MedImg(23), No. 7, July 2004, pp. 891-895.
IEEE Abstract. IEEE Top Reference. 0407
See also Ideal observer approximation using bayesian classification neural networks. BibRef

Edwards, D.C., Metz, C.E., Nishikawa, R.M.,
The Hypervolume Under the ROC Hypersurface of 'Near-Guessing' and 'Near-Perfect' Observers in N-Class Classification Tasks,
MedImg(24), No. 3, March 2005, pp. 293-299.
IEEE Abstract. IEEE Top Reference. 0501
BibRef

Edwards, D.C., Metz, C.E.,
Restrictions on the three-class ideal observer's decision boundary lines,
MedImg(24), No. 12, December 2005, pp. 1566-1573.
IEEE DOI Link 0601
BibRef

Edwards, D.C., Metz, C.E.,
Optimization of Restricted ROC Surfaces in Three-Class Classification Tasks,
MedImg(26), No. 10, October 2007, pp. 1345-1356.
IEEE DOI Link 0711
BibRef

He, X., Metz, C.E., Tsui, B.M.W., Links, J.M., Frey, E.C.,
Three-Class ROC Analysis: A Decision Theoretic Approach Under the Ideal Observer Framework,
MedImg(25), No. 5, May 2006, pp. 571-581.
IEEE DOI Link 0605
BibRef

He, X., Frey, E.C.,
Three-Class ROC Analysis: The Equal Error Utility Assumption and the Optimality of Three-Class ROC Surface Using the Ideal Observer,
MedImg(25), No. 8, August 2006, pp. 979-986.
IEEE DOI Link 0608
BibRef

He, X.[Xin], Frey, E.C.,
The Meaning and Use of the Volume Under a Three-Class ROC Surface (VUS),
MedImg(27), No. 5, May 2008, pp. 577-588.
IEEE DOI Link 0711
BibRef

He, X., Frey, E.C.,
The Validity of Three-Class Hotelling Trace (3-HT) in Describing Three-Class Task Performance: Comparison of Three-Class Volume Under ROC Surface (VUS) and 3-HT,
MedImg(28), No. 2, February 2009, pp. 185-193.
IEEE DOI Link 0902
BibRef

He, X., Frey, E.C.,
An Optimal Three-Class Linear Observer Derived From Decision Theory,
MedImg(26), No. 1, January 2007, pp. 77-83.
IEEE DOI Link 0701
BibRef

He, X.[Xin], Caffo, B.S., Frey, E.C.,
Toward Realistic and Practical Ideal Observer (IO) Estimation for the Optimization of Medical Imaging Systems,
MedImg(27), No. 10, October 2008, pp. 1535-1543.
IEEE DOI Link 0810
BibRef

He, X., Song, X., Frey, E.C.,
Application of Three-Class ROC Analysis to Task-Based Image Quality Assessment of Simultaneous Dual-Isotope Myocardial Perfusion SPECT (MPS),
MedImg(27), No. 11, November 2008, pp. 1556-1567.
IEEE DOI Link 0811
BibRef

Baraldi, A., Bruzzone, L., Blonda, P.,
Quality Assessment of Classification and Cluster Maps Without Ground Truth Knowledge,
GeoRS(43), No. 4, April 2005, pp. 857-873.
IEEE Abstract. IEEE Top Reference. 0501
BibRef

Santos-Pereira, C.M.[Carla M.], Pires, A.M.[Ana M.],
On optimal reject rules and ROC curves,
PRL(26), No. 7, 15 May 2005, pp. 943-952.
WWW Version. 0506
BibRef

DeVore, M.D.,
Estimates of Error Probability for Complex Gaussian Channels with Generalized Likelihood Ratio Detection,
PAMI(27), No. 10, October 2005, pp. 1580-1591.
IEEE DOI Link 0509
Two-class hypothesis testing. BibRef

Khurd, P., Gindi, G.,
Decision strategies that maximize the area under the LROC curve,
MedImg(24), No. 12, December 2005, pp. 1626-1636.
IEEE DOI Link 0601
BibRef

Ahlqvist, O.[Ola], Gahegan, M.[Mark],
Probing the Relationship Between Classification Error and Class Similarity,
PhEngRS(71), No. 12, December 2005, pp. 1365-1374.
WWW Version. 0602
A method that predicts land-cover classification errors by using semantic similarity metrics derived from land-cover taxonomy definitions. BibRef

Landgrebe, T.C.W.[Thomas C.W.], Tax, D.M.J.[David M.J.], Paclík, P.[Pavel], Duin, R.P.W.[Robert P.W.],
The interaction between classification and reject performance for distance-based reject-option classifiers,
PRL(27), No. 8, June 2006, pp. 908-917.
WWW Version. Unseen classes; Reject-option; Model selection 0605
BibRef

Fawcett, T.[Tom],
ROC graphs with instance-varying costs,
PRL(27), No. 8, June 2006, pp. 882-891.
WWW Version. Cost-sensitive learning; Classifier evaluation 0605
BibRef

Everson, R.M.[Richard M.], Fieldsend, J.E.[Jonathan E.],
Multi-class ROC analysis from a multi-objective optimisation perspective,
PRL(27), No. 8, June 2006, pp. 918-927.
WWW Version. Evolutionary computation; Pareto optimality; Gini coefficient 0605
BibRef

Matei, B.C.[Bogdan C.], Meer, P.[Peter],
Estimation of Nonlinear Errors-in-Variables Models for Computer Vision Applications,
PAMI(28), No. 10, October 2006, pp. 1537-1552.
IEEE DOI Link 0609
BibRef
Earlier:
A General Method for Errors-in-Variables Problems in Computer Vision,
CVPR00(II: 18-25).
IEEE Abstract. IEEE Top Reference.
WWW Version. 0005
HEIV. All measurements are noisy. Related to Sampson, renormalization, numerical. See also HEIV based estimation. BibRef

Georgescu, B.[Bogdan],
HEIV based estimation,
OnlineSeptember, 2002. Code, HEIV.
WWW Version. Code related to above paper. See also Estimation of Nonlinear Errors-in-Variables Models for Computer Vision Applications. BibRef 0209

Landgrebe, T.C.W.[Thomas C.W.], Duin, R.P.W.[Robert P.W.],
Approximating the multiclass ROC by pairwise analysis,
PRL(28), No. 13, 1 October 2007, pp. 1747-1758.
WWW Version. 0709
ROC analysis; Multiclass ROC; Cost sensitive; Threshold optimisation BibRef

Waegeman, W.[Willem], de Baets, B.[Bernard], Boullart, L.[Luc],
ROC analysis in ordinal regression learning,
PRL(29), No. 1, 1 January 2008, pp. 1-9.
WWW Version. 0711
ROC analysis; Ranking; Ordinal regression; Unbalanced learning problems; Performance measures; Machine learning BibRef

Gallas, B.D.[Brandon D.], Pennello, G.A.[Gene A.], Myers, K.J.[Kyle J.],
Multireader multicase variance analysis for binary data,
JOSA-A(24), No. 12, December 2007, pp. B70-B80.
WWW Version. 0801
Analyzing ROC (receiver operating characteristic) curve data. BibRef

Park, S., Badano, A., Gallas, B.D., Myers, K.J.,
Incorporating Human Contrast Sensitivity in Model Observers for Detection Tasks,
MedImg(28), No. 3, March 2009, pp. 339-347.
IEEE DOI Link 0903
BibRef

Marrocco, C.[Claudio], Duin, R.P.W., Tortorella, F.[Francesco],
Maximizing the area under the ROC curve by pairwise feature combination,
PR(41), No. 6, June 2008, pp. 1961-1974.
WWW Version. 0802
Two-class problems; ROC curve; Ranking; AUC BibRef

El Ayadi, M.M.H.[Moataz M.H.], Kamel, M.S.[Mohamed S.], Karray, F.[Fakhri],
Toward a tight upper bound for the error probability of the binary Gaussian classification problem,
PR(41), No. 6, June 2008, pp. 2120-2132.
WWW Version. 0802
Binary classification; Bayesian decision rule; Decision boundary; Error probability; Monte-Carlo simulations; Multivariate normal distribution; Quadratic surfaces BibRef

Chen, D.M.[Dong Mei], Wei, H.[Hui],
The effect of spatial autocorrelation and class proportion on the accuracy measures from different sampling designs,
PandRS(64), No. 2, March 2009, pp. 140-150.
Elsevier DOI Link
WWW Version. 0903
Accuracy assessment; Classification error; Sampling; Spatial autocorrelation; Class proportion BibRef

Paclik, P.[Pavel], Lai, C.[Carmen], Novovicova, J.[Jana], Duin, R.P.W.[Robert P.W.],
Variance estimation for two-class and multi-class ROC analysis using operating point averaging,
ICPR08(1-4).
IEEE DOI Link 0812
BibRef

Padmaja, T.M.[T. Maruthi], Dhulipalla, N.[Narendra], Krishna, P.R.[P. Radha], Bapi, R.S.[Raju S.], Laha, A.,
An Unbalanced Data Classification Model Using Hybrid Sampling Technique for Fraud Detection,
PReMI07(341-348).
Springer DOI Link 0712
BibRef

Fisher, R.B.,
An Empirical Model for Saturation and Capacity in Classifier Spaces,
ICPR06(IV: 189-193).
WWW Version. 0609
Determine the achievable classification rate for a database given a level of noise. BibRef

Maloof, M.A.,
On machine learning, ROC analysis, and statistical tests of significance,
ICPR02(II: 204-207).
IEEE DOI Link 0211
BibRef

Johnson, A.Y., Bobick, A.F.,
Relationship between identification metrics: Expected confusion and area under a ROC curve,
ICPR02(III: 662-666).
IEEE DOI Link 0211
BibRef

Rees, G.S., Wright, W.A., Greenway, P.,
ROC Method for the Evaluation of Multi-class Segmentation/Classification Algorithms with Infrared Imagery,
BMVC02(Poster Session). 0208
BibRef

Ménard, M., Doget, T., Shahin, A.,
Ambiguity Concept and Switching Regression Models,
SCIA99(Pattern Recognition). BibRef 9900

Raudys, S.J., Diciunas, V.,
Expected Error of Minimum Empirical Error and Maximal Margin Classifiers,
ICPR96(II: 875-879).
IEEE DOI Link 9608
(Institute of Mathematics and Informatics, LIT) BibRef

Kanungo, T., Gay, D.M., Haralick, R.M.,
Constrained monotone regression of ROC curves and histograms using splines and polynomials,
ICIP95(II: 292-295).
IEEE DOI Link 9510
BibRef

Grossman, T., Lapedes, A.,
Noise sensitivity signatures for model selection,
ICPR94(B:213-218).
IEEE DOI Link 9410
BibRef

Chapter on Pattern Recognition, Clustering, Statistics, Grammars, Learning, Neural Nets, Genetic Algorithms continues in
Multiple Classifiers, Combining Classifiers, Combinations .