Conf.RdCalculates a cross-tabulation of observed and predicted classes with associated statistics.
Conf(x, ...)
# S3 method for class 'table'
Conf(x, pos = NULL, ...)
# S3 method for class 'matrix'
Conf(x, pos = NULL, ...)
# Default S3 method
Conf(x, ref, pos = NULL, na.rm = TRUE, ...)
# S3 method for class 'rpart'
Conf(x, ...)
# S3 method for class 'multinom'
Conf(x, ...)
# S3 method for class 'glm'
Conf(x, cutoff = 0.5, pos = NULL, ...)
# S3 method for class 'randomForest'
Conf(x, ...)
# S3 method for class 'svm'
Conf(x, ...)
# S3 method for class 'regr'
Conf(x, ...)
# S3 method for class 'Conf'
plot(x, main = "Confusion Matrix", ...)
# S3 method for class 'Conf'
print(x, digits = max(3, getOption("digits") - 3), ...)
Sens(x, ...)
Spec(x, ...)a vector, normally a factor, of predicted classes or an object of following classes rpart, randomForest, svm, C50, glm, multinom, regr, lda, qda or table, resp. matrix. When a model is given, the predicted classes will be determined. A table or a matrix will be interpreted as a confusion matrix.
a vector, normally a factor, of classes to be used as the reference. This is ignored if x is a table or matrix.
a character string that defines the factor level corresponding to the "positive" results. Will be ignored for a \(n \times n\) table n > 2.
used in logit models. The cutoff for changing classes.
overall title for the plot.
controls the number of digits to print.
a logical value indicating whether or not missing values should be removed. Defaults to FALSE.
further arguments to be passed to or from methods.
The functions require the factors to have the same levels.
For two class problems, the sensitivity, specificity, positive
predictive value and negative predictive value is calculated using the
positive argument. Also, the prevalence of the "event" is computed from the
data (unless passed in as an argument), the detection rate (the rate of true events also
predicted to be events) and the detection prevalence (the prevalence of predicted events).
Suppose a \(2 \times 2\) table with notation
| Reference | ||
| Predicted | Event | No Event |
| Event | A | B |
| No Event | C | D |
The formulas used here are: $$Sensitivity = A/(A+C)$$ $$Specificity = D/(B+D)$$ $$Prevalence = (A+C)/(A+B+C+D)$$ $$PPV = (sensitivity * Prevalence)/((sensitivity*Prevalence) + ((1-specificity)*(1-Prevalence)))$$ $$NPV = (specificity * (1-Prevalence))/(((1-sensitivity)*Prevalence) + ((specificity)*(1-Prevalence)))$$ $$Detection Rate = A/(A+B+C+D)$$ $$Detection Prevalence = (A+B)/(A+B+C+D)$$ $$F-val Accuracy = 2 / (1/PPV + 1/Sensitivity)$$ $$Matthews Cor.-Coef = (A*D-B*C)/sqrt((A+B)*(A+C)*(D+B)*(D+C)) $$
See the references for discusions of the first five formulas.
For more than two classes, these results are calculated comparing each factor level to the remaining levels (i.e. a "one versus all" approach).
The overall accuracy and unweighted Kappa statistic are calculated. A p-value from McNemar's test is also computed using mcnemar.test (which can produce NA values with sparse tables).
The overall accuracy rate is computed along with a 95 percent confidence interval for this rate (using BinomCI) and a one-sided test to see if the accuracy is better than the "no information rate," which is taken to be the largest class percentage in the data.
The sensitivity is defined as the proportion of positive results out of the number of
samples which were actually positive. When there are no positive results, sensitivity is
not defined and a value of NA is returned. Similarly, when there are no negative
results, specificity is not defined and a value of NA is returned. Similar
statements are true for predictive values.
Confidence intervals for sensitivity, specificity etc. could be calculated as binomial confidence intervals (see BinomCI). BinomCI(A, A+C) yields the ci for sensitivity.
a list with elements
the results of table on data and reference
the positive result level
a numeric vector with overall accuracy and Kappa statistic values
the sensitivity, specificity, positive predictive value, negative predictive value, prevalence, dection rate and detection prevalence for each class. For two class systems, this is calculated once using the positive argument
Kuhn, M. (2008) Building predictive models in R using the caret package Journal of Statistical Software, (https://www.jstatsoft.org/v28/i05/).
Powers, David M W (2011) Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation (PDF). Journal of Machine Learning Technologies 2 (1): 37-63.
Collett D (1999) Modelling Binary Data. Chapman & Hall/CRC, Boca Raton Florida, pp. 24.
Matthews, B. W. (1975) Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure 405 (2): 442-451. doi:10.1016/0005-2795(75)90109-9. PMID 1180967.
# let tab be a confusion table
tab <- TextToTable("
lo hi
lo 23 13
hi 10 18 ", dimnames=c("pred", "obs"))
Conf(tab, pos="hi")
#>
#> Confusion Matrix and Statistics
#>
#> obs
#> pred hi lo
#> hi 18 10
#> lo 13 23
#>
#> Total n : 64
#> Accuracy : 0.6406
#> 95% CI : (0.5182, 0.7471)
#> No Information Rate : 0.5156
#> P-Value [Acc > NIR] : 0.0297
#>
#> Kappa : 0.2784
#> Mcnemar's Test P-Value : 0.6767
#>
#> Sensitivity : 0.5806
#> Specificity : 0.6970
#> Pos Pred Value : 0.6429
#> Neg Pred Value : 0.6389
#> Prevalence : 0.4844
#> Detection Rate : 0.4375
#> Detection Prevalence : 0.2812
#> Balanced Accuracy : 0.6388
#> F-val Accuracy : 0.6102
#> Matthews Cor.-Coef : 0.2797
#>
#> 'Positive' Class : hi
#>
pred <- Untable(tab)[,"pred"]
obs <- Untable(tab)[,"obs"]
Conf(x = pred, ref = obs)
#>
#> Confusion Matrix and Statistics
#>
#> Reference
#> Prediction lo hi
#> lo 23 13
#> hi 10 18
#>
#> Total n : 64
#> Accuracy : 0.6406
#> 95% CI : (0.5182, 0.7471)
#> No Information Rate : 0.5156
#> P-Value [Acc > NIR] : 0.0297
#>
#> Kappa : 0.2784
#> Mcnemar's Test P-Value : 0.6767
#>
#> Sensitivity : 0.6970
#> Specificity : 0.5806
#> Pos Pred Value : 0.6389
#> Neg Pred Value : 0.6429
#> Prevalence : 0.5156
#> Detection Rate : 0.5625
#> Detection Prevalence : 0.3594
#> Balanced Accuracy : 0.6388
#> F-val Accuracy : 0.6667
#> Matthews Cor.-Coef : 0.2797
#>
#> 'Positive' Class : lo
#>
Conf(x = pred, ref = obs, pos="hi")
#>
#> Confusion Matrix and Statistics
#>
#> Reference
#> Prediction hi lo
#> hi 18 10
#> lo 13 23
#>
#> Total n : 64
#> Accuracy : 0.6406
#> 95% CI : (0.5182, 0.7471)
#> No Information Rate : 0.5156
#> P-Value [Acc > NIR] : 0.0297
#>
#> Kappa : 0.2784
#> Mcnemar's Test P-Value : 0.6767
#>
#> Sensitivity : 0.5806
#> Specificity : 0.6970
#> Pos Pred Value : 0.6429
#> Neg Pred Value : 0.6389
#> Prevalence : 0.4844
#> Detection Rate : 0.4375
#> Detection Prevalence : 0.2812
#> Balanced Accuracy : 0.6388
#> F-val Accuracy : 0.6102
#> Matthews Cor.-Coef : 0.2797
#>
#> 'Positive' Class : hi
#>
Sens(tab) # Sensitivity
#> [1] 0.6969697
Spec(tab) # Specificity
#> [1] 0.5806452
tab <- TextToTable("
terrible poor marginal clear
terrible 10 4 1 0
poor 5 10 12 2
marginal 2 4 12 5
clear 0 2 6 13
", dimnames=c("pred", "obs"))
Conf(tab)
#>
#> Confusion Matrix and Statistics
#>
#> obs
#> pred terrible poor marginal clear
#> terrible 10 4 1 0
#> poor 5 10 12 2
#> marginal 2 4 12 5
#> clear 0 2 6 13
#>
#> Overall Statistics
#>
#> Total n : 88
#> Accuracy : 0.5114
#> 95% CI : (0.4087, 0.6131)
#> No Information Rate : 0.3523
#> P-Value [Acc > NIR] : 0.0016
#>
#> Kappa : 0.3449
#> Mcnemar's Test P-Value : NA
#>
#>
#> Statistics by Class:
#>
#> terrible poor marginal clear
#> Sensitivity 0.5882 0.5000 0.3871 0.6500
#> Specificity 0.9296 0.7206 0.8070 0.8824
#> Pos Pred Value 0.6667 0.3448 0.5217 0.6190
#> Neg Pred Value 0.9041 0.8305 0.7077 0.8955
#> Prevalence 0.1932 0.2273 0.3523 0.2273
#> Detection Rate 0.1705 0.3295 0.2614 0.2386
#> Detection Prevalence 0.1136 0.1136 0.1364 0.1477
#> Balanced Accuracy 0.7589 0.6103 0.5971 0.7662
#> F-val Accuracy 0.6250 0.4082 0.4444 0.6341
#> Matthews Cor.-Coef 0.5436 0.1967 0.2110 0.5234
#>