Guetzkow's U

So even though I couldn't do it because my coders couldn't agree, Guetzkow's U seemed like a useful way to measure intercoder reliability for frequency. I googled it and couldn't find the formula online, so today's post remedies that problem. For more information, see Folger et al. (1984) or Guetzkow (1950).

First, a caveat. Guetzkow's U statistic doesn't take chance into account, so treat it cautiously at best. This probably shouldn't be used alone, but only with other agreement statistics to provide an overall picture. Second, U doesn't count the agreement of specific codes but counts the agreement of number of codes. That works perfectly if there's a finite number of codes (the data's already unitized) and the codes are fairly simple. Bottomline, use common sense--this statistic could let you get away with poor coding, but doing so jeopardizes that anyone will take your results seriously.

U = Sum CodesA - Sum CodesB / Sum CodesA + Sum CodesB

Basically, U divides the difference in the number of codes between coders A and B by the total number of codes identified by A and B. If there's perfect agreement in the number, U will be 0. It's a measure of disagreement rather than agreement, so lower is better.

Again, remember, this statistic should never be used alone. In my summer project, I had good levels with one coder (U = .04) and poor levels with all other coders. However, three coders had decent levels with each other (U ~ .07) because they were all equally poor. This statistic can be very deceptive if used alone, but it focuses on freqency more than kappa or any other statistic.

I'm using a different version of kappa by Brennan and Prediger (1981) that considers marginals in a more forgiving way than does Cohen. I'm also using Krippendorf's alpha, and the SPSS macro for that is at Andrew Hayes' webpage at Ohio State University.
References:

Brennan, R. L. &; D. J. Prediger (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.

Folger, J.P., Hewes, D.E., & Poole, M.S. (1984). Coding social interaction. In B. Dervin & M. J. Voigt (Eds.), Progress in communication sciences (Vol. 4, pp. 115-161). Norwood, NJ: Ablex.

Guetzkow, H. (1950). Unitizing and categorizing problems in coding quantitative data. Journal of Clinical Psychology, 6, 47-58.