14.1.5.2.2 Classifier Combination, Evaluation, Overview, Appliction Specific

Boland, P.J.,
Majority Systems and the Condorcet Jury Theorem,
Statistician(38), 1989, pp. 181-189. For independent classifiers, error eat less than .5, for an odd number of classifiers, majority voting increases the correct decision rate as the number of classifiers increases. BibRef 8900

Xu, L., Krzyzak, A., Suen, C.Y.,
Methods of Combining Multiple Classifiers and Their Applications to Handwriting Recognition,
SMC(22), No. 3, 1992, pp. 418-435. Majority Voting for combinations. Unanimous consensus. Threshold Voting. Averaged Bayes Classifier. Dempster-Shafer. BibRef 9200

Xu, L.[Lei], Krzyzak, A., Oja, E.,
Unsupervised and supervised classifications by rival penalized competitive learning,
ICPR92(II:496-499).
IEEE DOI Link 9208
BibRef

Lam, L., Suen, C.Y.,
Application of Majority Voting to Pattern Recognition: An Analysis of Its Behavior and Performance,
SMC(27), No. 5, 1997, pp. 553-568. BibRef 9700
Earlier:
A Theoretical Analysis of the Application of Majority Voting to Pattern Recognition,
ICPR94(B:418-420).
IEEE DOI Link BibRef

Smyth, P.,
Bounds on the Mean Classification Error Rate of Multiple Experts,
PRL(17), No. 12, October 25 1996, pp. 1253-1257. 9612
BibRef

Kittler, J.V.[Joseph V.],
Combining Classifiers: A Theoretical Framework,
PAA(1), No. 1, 1998, pp. 18-27. BibRef 9800

Kittler, J.V., Hatef, M., Duin, R.P.W., Matas, J.,
On Combining Classifiers,
PAMI(20), No. 3, March 1998, pp. 226-239.
IEEE Abstract.
WWW Version. 9805
The combination rule using the most restrictive assumptions, the sum rule, did best. Compared versions of Max, Median, Majority vote rule dreived from See also Improving Model Accuracy Using Optimal Linear Combinations of Trained Neural Networks. Min rule derived from See also Multistage Algorithm for Fast Classification of Patterns, A. Also Sum and Product rules. BibRef

Kittler, J.V., Duin, R.P.W., Hatef, M.,
Combining Classifiers,
ICPR96(II: 897-901).
IEEE DOI Link 9608
(Univ. of Surrey, UK) BibRef

Wang, L.C., Der, S.Z., Nasrabadi, N.M.,
Composite Classifiers for Automatic Target Recognition,
OptEng(37), No. 3, March 1998, pp. 858-868. 9804
BibRef

Wang, L.C., Chan, L., Nasrabadi, N.M., and Der, S.Z.,
Combination of Two Learning Algorithms for Automatic Target Recognition,
ICIP97(I: 881-884).
IEEE DOI Link BibRef 9700

Ng, G.S., Singh, H.,
Data Equalization with Evidence Combination for Pattern Recognition,
PRL(19), No. 3-4, March 1998, pp. 227-235. 9807
BibRef

Tax, D.M.J.[David M.J.], van Breukelen, M.[Martijn], Duin, R.P.W.[Robert P.W.], Kittler, J.V.[Josef V.],
Combining multiple classifiers by averaging or by multiplying?,
PR(33), No. 9, September 2000, pp. 1475-1485.
WWW Version. 0005
BibRef

van Breukelen, M.[Martijn], Duin, R.P.W.[Robert P.W.],
Neural Network Initialization by Combined Classifiers,
ICPR98(Vol I: 215-218).
IEEE DOI Link 9808
BibRef

Pudil, P., Novovicova, J., Blaha, S., Kittler, J.V.,
Multistage Pattern Recognition with Reject Option,
ICPR92(II:92-95).
IEEE DOI Link BibRef 9200

Kittler, J.V., Hojjatoleslami, A., Windeatt, T.,
Strategies for Combining Classifiers Employing Shared and Distinct Pattern Representations,
PRL(18), No. 11-13, November 1997, pp. 1373-1377. 9806
BibRef
Earlier:
Weighting Factors in Multiple Expert Fusion,
BMVC97(xx-yy).
HTML Version. 0209
BibRef

Hojjatoleslami, A., Kittler, J.V.[Josef V.],
Strategies for Weighted Combination of Classifiers Employing Shared and Distinct Pattern Representations,
ICPR98(Vol I: 338-340).
IEEE DOI Link 9808
BibRef

Kittler, J.V.[Josef V.], Hojjatoleslami, A.[Ali],
A Weighted Combination of Classifiers Employing Shared and Distinct Representations,
CVPR98(924-929).
IEEE Abstract. BibRef 9800

Prior, M., Windeatt, T.,
Parameter Tuning using the Out-of-Bootstrap Generalisation Error Estimate for Stochastic Discrimination and Random Forests,
ICPR06(II: 498-501).
WWW Version. 0609
BibRef

Alkoot, F.M., Kittler, J.V.,
Experimental Evaluation of Expert Fusion Strategies,
PRL(20), 1999, pp. 1361-1369. BibRef 9900
And:
Improving the Performance of the Product Fusion Strategy,
ICPR00(Vol II: 164-167).
IEEE DOI Link 0009
Compared Minimum, Maximum, Average, Median, Majority. Futher analysis in: See also Theoretical Study on Six Classifier Fusion Strategies, A. BibRef

Kittler, J.V., Alkoot, F.M.,
Sum versus vote fusion in multiple classifier systems,
PAMI(25), No. 1, January 2003, pp. 110-115.
IEEE Abstract.
WWW Version. 0301
Analysis of fusion rules. BibRef

Martínez Trinidad, J.F.[José Francisco], Guzmán Arenas, A.[Adolfo],
The logical combinatorial approach to pattern recognition, an overview through selected works,
PR(34), No. 4, April 2001, pp. 741-751.
WWW Version. 0101
BibRef

Kuncheva, L.I.[Ludmila I.],
Switching between selection and fusion in combining classifiers: An Experiment,
SMC-B(32), No. 2, April 2002, pp. 146-156.
IEEE Top Reference. 0205
BibRef

Kuncheva, L.I.[Ludmila I.],
Using diversity measures for generating error-correcting output codes in classifier ensembles,
PRL(26), No. 1, 1 January 2005, pp. 83-90.
WWW Version. 0501
BibRef

Kuncheva, L.I.[Ludmila I.],
A Theoretical Study on Six Classifier Fusion Strategies,
PAMI(24), No. 2, February 2002, pp. 281-286.
IEEE Abstract.
WWW Version. 0202
2 classes and L classifiers. Minimum, Maximum, Average, Median, Majority, Product. Minimum/Maximum best. See also Experimental Evaluation of Expert Fusion Strategies. BibRef

Kuncheva, L.I., and Whitaker, C.J.,
Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy,
MachLearn(51), 2003, pp. 181-207 BibRef 0300

Kuncheva, L.I., Whitaker, C.J., Shipp, C.A., Duin, R.P.W.,
Limits on the Majority Vote Accuracy in Classifier Fusion,
PAA(6), No. 1, 2003, pp. 22-31. BibRef 0300
Earlier:
Is Independence Good for Combining Classifiers?,
ICPR00(Vol II: 168-171).
IEEE DOI Link 0009
BibRef

Kuncheva, L.I., Whitaker, C.J., Narasimhamurthy, A.,
A case-study on naive labelling for the nearest mean and the linear discriminant classifiers,
PR(41), No. 10, October 2008, pp. 3010-3020.
WWW Version. 0808
Semi-supervised learning; Unlabelled data; On-line classifiers; Naive labelling BibRef

Alexandre, L.A.[Luís A.], Campilho, A.C.[Aurélio C.], Kamel, M.S.[Mohamed S.],
On combining classifiers using sum and product rules,
PRL(22), No. 12, October 2001, pp. 1283-1289.
Elsevier DOI Link 0108
BibRef
Earlier:
Combining Independent and Unbiased Classifiers Using Weighted Average,
ICPR00(Vol II: 495-498).
IEEE DOI Link 0009
BibRef

Lu, Y.[Yue], Tan, C.L.[Chew Lim],
Combination of multiple classifiers using probabilistic dictionary and its application to postcode recognition,
PR(35), No. 12, December 2002, pp. 2823-2832.
WWW Version. 0209
BibRef

Oh, S.B.[Sang-Bong],
On the relationship between majority vote accuracy and dependency in multiple classifier systems,
PRL(24), No. 1-3, January 2003, pp. 359-363.
Elsevier DOI Link 0211
BibRef

Murua, A.[Alejandro],
Upper Bounds for Error Rates of Linear Combinations of Classifiers,
PAMI(24), No. 5, May 2002, pp. 591-602.
IEEE Abstract.
WWW Version. 0205
Analyze classifiers constructed with same training data. Depending on dependence between them, linear combinations will achieve good error performance. BibRef

Giacinto, G.[Giorgio], Roli, F.[Fabio],
An approach to the automatic design of multiple classifier systems,
PRL(22), No. 1, January 2001, pp. 25-33.
Elsevier DOI Link 0105
See also Combination of neural and statistical algorithms for supervised classification of remote-sensing images. BibRef
Earlier:
A Theoretical Framework for Dynamic Classifier Selection,
ICPR00(Vol II: 8-11).
IEEE DOI Link 0009
BibRef
Earlier:
Methods for dynamic classifier selection,
CIAP99(659-664).
IEEE DOI Link 9909
BibRef
Earlier:
Adaptive selection of image classifiers,
CIAP97(I: 38-45).
WWW Version. 9709
BibRef

Fumera, G.[Giorgio], Roli, F.[Fabio], Giacinto, G.[Giorgio],
Reject option with multiple thresholds,
PR(33), No. 12, December 2000, pp. 2099-2101.
WWW Version. 0401
BibRef

Pillai, I.[Ignazio], Fumera, G.[Giorgio], Roli, F.[Fabio],
A Classification Approach with a Reject Option for Multi-label Problems,
CIAP11(I: 98-107).
Springer DOI Link 1109
BibRef

Giacinto, G.[Giorgio], Roli, F.[Fabio],
Dynamic classifier selection based on multiple classifier behaviour,
PR(34), No. 9, September 2001, pp. 1879-1881.
WWW Version. 0108
BibRef

Giacinto, G.[Giorgio], Roli, F.[Fabio], Didaci, L.[Luca],
Fusion of multiple classifiers for intrusion detection in computer networks,
PRL(24), No. 12, August 2003, pp. 1795-1803.
WWW Version. 0304
BibRef
Earlier: A1, A2 only:
Intrusion detection in computer networks by multiple classifier systems,
ICPR02(II: 390-393).
IEEE DOI Link 0211
BibRef

Giacinto, G.[Giorgio], Perdisci, R.[Roberto], Roli, F.[Fabio],
Network Intrusion Detection by Combining One-Class Classifiers,
CIAP05(58-65).
Springer DOI Link 0509
BibRef

Fumera, G.[Giorgio], Roli, F.[Fabio],
Analysis of error-reject trade-off in linearly combined multiple classifiers,
PR(37), No. 6, June 2004, pp. 1245-1265.
WWW Version. 0405
BibRef

Roli, F., Fumera, G., Vernazza, G.,
Analysis of error-reject trade-off in linearly combined classifiers,
ICPR02(II: 120-123).
IEEE DOI Link 0211
BibRef
Earlier: A2, A1, A3:
A method for error rejection in multiple classifier systems,
CIAP01(454-458).
IEEE Top Reference. 0210
BibRef

Lin, X.F.[Xiao-Fan], Yacoub, S.[Sherif], Burns, J.[John], Simske, S.J.[Steven J.],
Performance analysis of pattern classifier combination by plurality voting,
PRL(24), No. 12, August 2003, pp. 1959-1969.
WWW Version. 0304
BibRef

Fumera, G.[Giorgio], Roli, F.[Fabio],
A Theoretical and Experimental Analysis of Linear Combiners for Multiple Classifier Systems,
PAMI(27), No. 6, June 2005, pp. 942-956.
IEEE Abstract. 0505
Analysis follows on See also Analysis of decision boundaries in linearly combined neural classifiers. Performance depends on individual classifiers and correlation between them. BibRef

Fumera, G.[Giorgio], Fabio, R.[Roli], Alessandra, S.[Serrau],
A Theoretical Analysis of Bagging as a Linear Combination of Classifiers,
PAMI(30), No. 7, July 2008, pp. 1293-1299.
IEEE DOI Link 0806
Analysis derived from linear combinations to Bagging approaches. BibRef

Biggio, B.[Battista], Fumera, G.[Giorgio], Roli, F.[Fabio],
Adversarial Pattern Classification Using Multiple Classifiers and Randomisation,
SSPR08(500-509).
Springer DOI Link 0812
BibRef

Didaci, L.[Luca], Giacinto, G.[Giorgio], Roli, F.[Fabio], Marcialis, G.L.[Gian Luca],
A study on the performances of dynamic classifier selection based on local accuracy estimation,
PR(38), No. 11, November 2005, pp. 2188-2191.
WWW Version. 0509
BibRef

Yildiz, O.T.[Olcay Taner], Alpaydin, E.[Ethem],
Ordering and Finding the Best of K>2 Supervised Learning Algorithms,
PAMI(28), No. 3, March 2006, pp. 392-402.
IEEE DOI Link 0602
Given a dataset and a set of algorithms, find the one that is best. BibRef

Vilariño, F.[Fernando], Kuncheva, L.I.[Ludmila I.], Radeva, P.I.[Petia I.],
ROC curves and video analysis optimization in intestinal capsule endoscopy,
PRL(27), No. 8, June 2006, pp. 875-881.
WWW Version. Classifiers ensemble; Imbalanced classes; Wireless capsule endoscopy 0605
BibRef

Rodriguez, J.J., Kuncheva, L.I.[Ludmila I.], Alonso, C.J.,
Rotation Forest: A New Classifier Ensemble Method,
PAMI(28), No. 10, October 2006, pp. 1619-1630.
IEEE DOI Link 0609
BibRef

Rodríguez, J.J.[Juan J.], Kuncheva, L.I.[Ludmila I.],
Combining Online Classification Approaches for Changing Environments,
SSPR08(520-529).
Springer DOI Link 0812
BibRef

Kuncheva, L.I.[Ludmila I.], Vetrov, D.P.,
Evaluation of Stability of k-Means Cluster Ensembles with Respect to Random Initialization,
PAMI(28), No. 11, November 2006, pp. 1798-1808.
IEEE DOI Link 0609
BibRef

Cabrera, J.B.D.[João B.D.],
On the impact of fusion strategies on classification errors for large ensembles of classifiers,
PR(39), No. 11, November 2006, pp. 1963-1978.
WWW Version. 0608
Classifier fusion; Asymptotic methods; Independent classifiers; Sensor networks BibRef

Canuto, A.M.P.[Anne M.P.], Abreu, M.C.C.[Marjory C.C.], de Melo Oliveira, L.[Lucas], Xavier, Jr., J.C.[João C.], de M. Santos, A.[Araken],
Investigating the influence of the choice of the ensemble members in accuracy and diversity of selection-based and fusion-based methods for ensembles,
PRL(28), No. 4, 1 March 2007, pp. 472-486.
WWW Version. 0701
Diversity measures; Classifier ensembles; Selection-based combination methods; Fusion-based combination methods BibRef

Hu, R.[Roland], Damper, R.I.,
A 'No Panacea Theorem' for classifier combination,
PR(41), No. 8, August 2008, pp. 2665-2673.
WWW Version. 0805
BibRef
Earlier:
A 'No Panacea Theorem' for Multiple Classifier Combination,
ICPR06(II: 1250-1253).
WWW Version. 0609
Probability density functions; Gaussian mixtures; `No Free Lunch' theorems BibRef

García, V.[Vicente], Sánchez, J.S.[Javier Salvador], Mollineda, R.A.[Ramón A.],
Classification of High Dimensional and Imbalanced Hyperspectral Imagery Data,
IbPRIA11(644-651).
Springer DOI Link 1106
BibRef
Earlier: A1, A3, A2:
Theoretical Analysis of a Performance Measure for Imbalanced Data,
ICPR10(617-620).
IEEE DOI Link 1008
BibRef
Earlier: A1, A3, A2:
Index of Balanced Accuracy: A Performance Measure for Skewed Class Distributions,
IbPRIA09(441-448).
Springer DOI Link 0906
BibRef
And: A1, A3, A2:
A New Performance Evaluation Method for Two-Class Imbalanced Problems,
SSPR08(917-925).
Springer DOI Link 0812
BibRef
Earlier: A1, A2, A3:
An Empirical Study of the Behavior of Classifiers on Imbalanced and Overlapped Data Sets,
CIARP07(397-406).
Springer DOI Link 0711
BibRef

García, V., Mollineda, R.A., Sánchez, J.S., Alejo, R., Martínez Sotoca, J.[José],
When Overlapping Unexpectedly Alters the Class Imbalance Effects,
IbPRIA07(II: 499-506).
Springer DOI Link 0706
BibRef

Freitas, C.O.A.[Cinthia O. A.], de Carvalho, J.M.[João M.], Oliveira, J.J.[José-Josemar], Aires, S.B.K.[Simone B. K.], Sabourin, R.[Robert],
Confusion Matrix Disagreement for Multiple Classifiers,
CIARP07(387-396).
Springer DOI Link 0711
BibRef

Moreno-Seco, F.[Francisco], Iñesta, J.M.[José M.], Ponce de León, P.J.[Pedro J.], Micó, L.[Luisa],
Comparison of Classifier Fusion Methods for Classification in Pattern Recognition Tasks,
SSPR06(705-713).
Springer DOI Link 0608
BibRef

Abdul Kader, A.[Ahmad], Drakopoulos, J.A.[John A.], Zhang, Q.[Qi],
Comparative Classifier Aggregation,
ICPR06(III: 156-159).
WWW Version. 0609
BibRef

Bertolami, R.[Roman], Bunke, H.[Horst],
Early feature stream integration versus decision level combination in a multiple classifier system for text line recognition,
ICPR06(II: 845-848).
WWW Version. 0609
BibRef

Duin, R.P.W.,
The combining classifier: to train or not to train?,
ICPR02(II: 765-770).
IEEE DOI Link 0211
BibRef

Schiele, B.,
How many classifiers do I need?,
ICPR02(II: 176-179).
IEEE DOI Link 0211
BibRef

Chapter on Pattern Recognition, Clustering, Statistics, Grammars, Learning, Neural Nets, Genetic Algorithms continues in
Decision Fusion .