Krippendorff's Alpha Reliability Coefficient

Calculate the alpha coefficient of reliability proposed by Krippendorff.

KrippAlpha(x, method=c("nominal", "ordinal", "interval", "ratio"))

Arguments

x

classifier x object matrix of classifications or scores

method

data level of x

Value

A list with class '"irrlist"' containing the following components:

method

a character string describing the method.

subjects

the number of data objects.

raters

the number of raters.

irr.name

a character string specifying the name of the coefficient.

value

value of alpha.

stat.name

here "nil" as there is no test statistic.

statistic

the value of the test statistic (NULL).

p.value

the probability of the test statistic (NULL).

cm

the concordance/discordance matrix used in the calculation of alpha

data.values

a character vector of the unique data values

levx

the unique values of the ratings

nmatchval

the count of matches, used in calculation

data.level

the data level of the ratings ("nominal","ordinal", "interval","ratio")

References

Krippendorff, K. (1980) Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage.

Author

Jim Lemon <jim@bitwrit.com.au>

Note

Krippendorff's alpha coefficient is particularly useful where the level of measurement of classification data is higher than nominal or ordinal. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-016-0200-9

Note

This function was previously published as kripp.alpha() in the irr package and has been integrated here without logical changes, but with some adaptations in the result structure.

Examples

# the "C" data from Krippendorff
 nmm <- matrix(c(1,1,NA,1,2,2,3,2,3,3,3,3,3,3,3,3,2,2,2,2,1,2,3,4,4,4,4,4,
                 1,1,2,1,2,2,2,2,NA,5,5,5,NA,NA,1,1,NA,NA,3,NA), nrow=4)
 
 # first assume the default nominal classification
 KrippAlpha(nmm)
#> $method
#> [1] "Krippendorff's alpha"
#> 
#> $subjects
#> [1] 12
#> 
#> $raters
#> [1] 4
#> 
#> $irr.name
#> [1] "alpha"
#> 
#> $value
#> [1] 0.7434211
#> 
#> $stat.name
#> [1] "nil"
#> 
#> $statistic
#> NULL
#> 
#> $cm
#>           [,1]       [,2]      [,3]      [,4] [,5]
#> [1,] 7.0000000  1.3333333 0.3333333 0.3333333    0
#> [2,] 1.3333333 10.0000000 1.3333333 0.3333333    0
#> [3,] 0.3333333  1.3333333 8.0000000 0.3333333    0
#> [4,] 0.3333333  0.3333333 0.3333333 4.0000000    0
#> [5,] 0.0000000  0.0000000 0.0000000 0.0000000    3
#> 
#> $data.values
#> [1] "1" "2" "3" "4" "5"
#> 
#> $nmatchval
#> [1] 40
#> 
#> $data.level
#> [1] "nominal"
#> 
#> attr(,"class")
#> [1] "irrlist"
 
 # now use the same data with the other three methods
 KrippAlpha(nmm, "ordinal")
#> $method
#> [1] "Krippendorff's alpha"
#> 
#> $subjects
#> [1] 12
#> 
#> $raters
#> [1] 4
#> 
#> $irr.name
#> [1] "alpha"
#> 
#> $value
#> [1] 0.8153875
#> 
#> $stat.name
#> [1] "nil"
#> 
#> $statistic
#> NULL
#> 
#> $cm
#>           [,1]       [,2]      [,3]      [,4] [,5]
#> [1,] 7.0000000  1.3333333 0.3333333 0.3333333    0
#> [2,] 1.3333333 10.0000000 1.3333333 0.3333333    0
#> [3,] 0.3333333  1.3333333 8.0000000 0.3333333    0
#> [4,] 0.3333333  0.3333333 0.3333333 4.0000000    0
#> [5,] 0.0000000  0.0000000 0.0000000 0.0000000    3
#> 
#> $data.values
#> [1] "1" "2" "3" "4" "5"
#> 
#> $nmatchval
#> [1] 40
#> 
#> $data.level
#> [1] "ordinal"
#> 
#> attr(,"class")
#> [1] "irrlist"
 KrippAlpha(nmm, "interval")
#> $method
#> [1] "Krippendorff's alpha"
#> 
#> $subjects
#> [1] 12
#> 
#> $raters
#> [1] 4
#> 
#> $irr.name
#> [1] "alpha"
#> 
#> $value
#> [1] 0.8491071
#> 
#> $stat.name
#> [1] "nil"
#> 
#> $statistic
#> NULL
#> 
#> $cm
#>           [,1]       [,2]      [,3]      [,4] [,5]
#> [1,] 7.0000000  1.3333333 0.3333333 0.3333333    0
#> [2,] 1.3333333 10.0000000 1.3333333 0.3333333    0
#> [3,] 0.3333333  1.3333333 8.0000000 0.3333333    0
#> [4,] 0.3333333  0.3333333 0.3333333 4.0000000    0
#> [5,] 0.0000000  0.0000000 0.0000000 0.0000000    3
#> 
#> $data.values
#> [1] "1" "2" "3" "4" "5"
#> 
#> $nmatchval
#> [1] 40
#> 
#> $data.level
#> [1] "interval"
#> 
#> attr(,"class")
#> [1] "irrlist"
 KrippAlpha(nmm, "ratio") 
#> $method
#> [1] "Krippendorff's alpha"
#> 
#> $subjects
#> [1] 12
#> 
#> $raters
#> [1] 4
#> 
#> $irr.name
#> [1] "alpha"
#> 
#> $value
#> [1] 0.7974028
#> 
#> $stat.name
#> [1] "nil"
#> 
#> $statistic
#> NULL
#> 
#> $cm
#>           [,1]       [,2]      [,3]      [,4] [,5]
#> [1,] 7.0000000  1.3333333 0.3333333 0.3333333    0
#> [2,] 1.3333333 10.0000000 1.3333333 0.3333333    0
#> [3,] 0.3333333  1.3333333 8.0000000 0.3333333    0
#> [4,] 0.3333333  0.3333333 0.3333333 4.0000000    0
#> [5,] 0.0000000  0.0000000 0.0000000 0.0000000    3
#> 
#> $data.values
#> [1] "1" "2" "3" "4" "5"
#> 
#> $nmatchval
#> [1] 40
#> 
#> $data.level
#> [1] "ratio"
#> 
#> attr(,"class")
#> [1] "irrlist"