Monthly Archives: March 2014

Percent Agreement IRR (for duration data)

In this post I will demonstrate how to calculate a percent agreement for IRR with some very simple scripting. This kind of IRR has an additional step of creating new rows in the timeline after combining both coders’ data. I do not know of a way to calculate “and” and “or” conditions of two rows within a scripting window to avoid this step (though if there is a way, I’d love to learn how…I tried to make up scripts myself, but they didn’t work ;-))

I made two videos today. In the first, I explain the behavioral scenario and demonstrate how to easily duplicate the code window (in my last post I just described it).

In this second video I demonstrate how to make the new rows after both coders data is on the same timeline (they can either code on the same timeline to begin with or they can code separately and combine timelines later).

Advertisements

Inter-Rater Reliability Series – Point-by-Point Agreement

I promise I will go back to my series on open-coding, but after the webinar I gave last week I realized there was a big interest in using Studiocode for Inter-rater reliability (IRR). There are several methods for calculating IRR, and the method you choose is dependent on your data type. In this first video I will give detailed instructions on how to calculate IRR for frequency-count data using point-by-point agreement. The video I am coding is another example of a teacher using Discrete trial training with a student. I coded the video for “correct responses” by the student. If this were a research study, and I were drawing conclusions about how this strategy improves a student’s accuracy, I would need to make sure I was accurately coding the student responses by using a second coder as a reliability checker. In this video I explain how I could do this with Studiocode.

Searching for Trends

The process of open coding is a laboring task (but if you are a qualitative researcher, you already know that). I spent some time adding labels into my “behavioral prompt” instances to start looking for trends with which I could start to develop my code window. I apologize in advance for some of my random thoughts that popped up as I was making this screencast…I was having a difficult time verbalizing my process because it was quite involved. I am hoping as you listen to my “think aloud” you might get some ideas on how to do this for your research.

The purpose of doing this open coding with text labels is not to do this for ALL of your videos. You will do this to look for trends to develop more codes and a structure for those codes. Your goal should be to eventually use your code window (which will contain both codes and labels) to code your videos. Depending on your research question, you may only need to do open coding for a video or two to come up with a solid code hierarchy. You can still add and modify your code window as you go, but the sooner you can have multiple codes and labels, the easier it will be for you to find patterns. 🙂

Problems Solved!

Thank you to Philip and Owen for their wonderful solutions. For those who didn’t see their comments or for those who need to see things in action rather than just read directions, I have demonstrated their solutions. 🙂

Adjusting Coded Instances

When we start coding instances, sometimes we make an error that we need to adjust later.  When I went back to my coded instances of “behavioral prompts” I realized I missed one between instances 2 and 3.  This is actually going to be a two-part post.  First I will show you how I corrected that problem, and then I will post a second video presenting some problems I was having with this process in hopes that someone from Studiocode can tell me what I was doing wrong.  Often there are many solutions to the same problem, but some are easier/simpler than others.

You probably noticed how the video above cut off rather abruptly…this is because I got stuck trying to figure out how to adjust the length of instances without adjusting all of the instances that followed.  The screencast below is where I explain my difficulties (perhaps some of my viewers have encountered similar difficulties).  When I figure out how to address my difficulties, I will make a new post 🙂