Kappa

---

Interobserver variability - inconsistency between observers (two radiologist reading a x-ray).

Intraobserver variability - inconsistency in a observer (single radiologist reading the same film on more than one occasion).

kappa - a measure of agreement between two observers taking into account agreement that could occur by chance (expected agreement).

kappa =	Observed agreement - Chance agreement
	--------------------------------------------------
	Total observed - Chance agreement

or using percentages

kappa =	Observed agreement - Expected agreement
	--------------------------------------------------
	100% - Expected agreement

Of note, kappa tells us nothing about the validity of the measurement.

You will not have to calculate kappa but an example may help you understand the concept.

		Exam 2
		Positive	Negative
Exam 1	Positive	a	b	a+b
	Negative	c	d	c+d
		a+c	b+d	a+b+c+d

Expected agreement for cell a =	(a+b) (a+c)
	----------------------------------------------
	(a+b+c+d)

Expected agreement for cell d =	(c+d) (b+d)
	----------------------------------------------
	(a+b+c+d)

Expected agreement = expected agreement for cell a + expected agreement for cell d

Observed agreement =	(a+d)
	-----------------------------------------------
	(a+b+c+d)

 

Two third year medical students are viewing chest X-rays (CXR) of HIV positive patients who are short of breath, looking for interstitial infiltrates suggestive of pneumocystis carinii pneumonia. The two students both agreed that 10 films showed interstitial infiltrates and agreed that 5 films were clear. They disagreed on 10 other films.

		Student 2		Student 1 totals
		Interstitial Infiltrates	Clear CXR
Student 1	Interstitial Infiltrates	10	5	15 positive
	Clear CXR	5	5	10 negative
Student 2 totals		15 positive	10 negative	25 total

The expected chance of both finding interstitial infiltrates = ((15) X (15)) / 25 = 9

The expected chance of both finding a clear CXR = ((10) X (10)) / 25 = 4

So the chance of the medical students agreeing are 9 + 4 = 13

The observed frequency of agreement is 10 + 5 = 15

kappa =	15 - 13	= 0.17
	----------------------------------------------
	25 - 13

Another example: Using the numbers from an article where two observers assessed the sonograms of all 220 study patients looking for deep venous thrombosis in their lower extremities.

		Observer 2		Observer 1 totals
		Noncompressible veins	Compressible veins
Observer 1	Noncompressible veins	71	0	71
	Compressible viens	0	149	149
Observer 2 totals		71	149	220 total

The expected chance of both observers finding noncpmpressible veins = ((71) X (71)) / 220 = 22.9

The expected chance of both observers finding compressible veins = ((149) X (149)) / 220 = 100.9

So the chance of the observers agreeing are 22.9 + 100.9 = 123.8

The observed frequency of agreement is 71+ 149 = 220

kappa =	220-123.8	= 1.0
	-----------------------------------------------
	220-123.8

 

Interpretation of kappa values*
kappa	Interpretation
<0	No agreement
0.0-0.19	Poor agreement
0.20-0.39	Fair agreement
0.40-0.59	Moderate agreement
0.60-0.79	Substantial agreement
0.80-1.00	Almost perfect agreement

*Landis, JR and Koch, GG. The measurement of observer agreement for categorical data. Biometrics 33:159-174, 1977.