I just wanted to write a quick update on my progress with IRR. I just met with Ryan and we discussed some ways for me to calculate all of the IRR measurements I want. Now I just need some time to play with my code windows and figure out for which values I want to calculate IRR and which formulas I should use for reporting. Ideally I would like to use more than percent agreement for some measures; a more appropriate calculation would be Cohen’s Kappa. Looking at the scripting involved for Kappa, it seems like this is much more complicated, but I am still going to give it a try. The great thing about using scripting to calculate this is that it will only be hard once! Once it’s in there, I can calculate Kappa for any videos we’ve coded. I will continue to develop the IRR portion of my Independence Code Windows over the next few days and then I will start working on my Engagement Code Window, which is much more complicated.
On a separate note, In the next few weeks I will train my research cohort on some basics of StudioCode and how we will use these code windows for our fall research. I plan on getting some feedback from them and posting it on this blog so you all can hear the perspectives of some other educational researchers 🙂