Kappa Calculation With 3 Categories

In this video I show you how to calculate Kappa with Studiocode when there are 3 category choices for instances.  Basically the main difference is in the matrix of agreement options.  The 4-cell matrix when there are 2 choices becomes a 9-cell matrix (see matrix below):

Screen Shot 2014-05-30 at 11.13.24 AM

The “agreement” cells are now cells a, e, and i.


 

Attached to this post I have included a new PDF document as a reference as well as a copy of the code window with the scripts already in them.  Feel free to copy and adapt these resources to fit your research needs.  This same pattern can be used for 4 or more category Kappa calculations.  HINT:  Make the matrix first and identify the agreement cells (the downward diagonal).

New Kappa Code Window:

https://drive.google.com/file/d/0B1mdRDzIaULfY2k5T05SbEo4UWM/edit?usp=sharing

3 Group Kappa Directions:

https://drive.google.com/file/d/0B1mdRDzIaULfZk83NVZpOFBnWms/edit?usp=sharing


 

As I mentioned in the video, make sure you are thinking about sample size (frequency of code), especially with larger matrices.  I would recommend calculating Kappa after you combine timelines for multiple occasions.  For infrequent codes, there are better IRR options.

Advertisements

Posted on May 30, 2014, in IRR. Bookmark the permalink. 1 Comment.

  1. Tara, another excellent post.

    I thought an interesting additional script line would be along these lines:

    if “K” < .2 show "slightly significant" etc.

    Regards
    Philip

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: