Monthly Archives: August 2013

Kappa vs. Point by Point Agreement

Today’s post is a little bit longer than my other posts because I had to add a little Statistics background before I discussed my IRR coding.

Quick Update

I just wanted to write a quick update on my progress with IRR.   I just met with Ryan and we discussed some ways for me to calculate all of the IRR measurements I want.  Now I just need some time to play with my code windows and figure out for which values I want to calculate IRR and which formulas I should use for reporting.  Ideally I would like to use more than percent agreement for some measures; a more appropriate calculation would be Cohen’s Kappa.  Looking at the scripting involved for Kappa, it seems like this is much more complicated, but I am still going to give it a try.  The great thing about using scripting to calculate this is that it will only be hard once! Once it’s in there, I can calculate Kappa for any videos we’ve coded.  I will continue to develop the IRR portion of my Independence Code Windows over the next few days and then I will start working on my Engagement Code Window, which is much more complicated.

On a separate note, In the next few weeks I will train my research cohort on some basics of StudioCode and how we will use these code windows for our fall research.  I plan on getting some feedback from them and posting it on this blog so you all can hear the perspectives of some other educational researchers 🙂                

More on IRR

I apologize for the long gaps between posts these past few weeks.  My new semester starts today and I had to finish up an online course, so my time to spend on my research/this blog was cut short.  I am ready to really dive back in now though 🙂

First off, I would like to address some of the wonderful comments on my last post:

1)   Mike – Thank you so much for the keyboard shortcut tips! I’ve been using a Mac for the past 8 years, and I still don’t have them all down.  😉

2)   Ryan – Thank you for the solution for Row A or Row B or Row C – This will work perfectly.  As you all have shown me, there are multiple ways to accomplish this.  I’m sure there will be times where one method is easier/better than another.

3)   Owen – You asked if I needed to see the row for Independent Working, and I’m not sure yet.  If I wind up using any text labels that attach to the Independent working code, then yes, I will need it for scripting.  As it is used now, I shouldn’t need it.  I’ve been slowly getting the hang of the scripting and the ability to use mathematics to calculate values in the scripts.

4)   Phillip – You raise an excellent point about some other ways to calculate IRR with more subjective rating systems.  In terms of measuring independence, I don’t need to calculate a magnitude of independence, but there are definitely some behavioral observations for which magnitude would be a very appropriate measurement.

Here is an example where both frequency, magnitude, and duration would need to be coded and evaluated for IRR:

If I were coding a student for aggressive behavior, I would want my raters to look both at frequency, magnitude, and duration of the behavior.  I would want to see if they marked the same incidents as “aggression” (the frequency IRR), I would want to compare the lengths of the incidents, and I would want to have operational definitions for different magnitudes of aggression (common magnitude measurements for aggression are: mild, moderate, and severe). I think I would use text labels for the magnitude categories so each incident would be labeled with the level.

For the duration, I would calculate IRR in the same way I calculated it in my last post.: (A & B)/(A or B) x 100.   For frequency and magnitude it gets a little trickier.  I can easily look at the timelines and compare incident by incident to determine if the same incidents were coded and if the magnitude levels match, but this doesn’t save me much time because I still have to analyze each list.  If I used that method, StudioCode doesn’t actually do anything different for me than a simple matrix or a checklist would.  Does anyone have any ideas for ways to use scripting to report out IRR for those measurements?  I’m going to think about this one and see what I come up with.  Right now I can easily figure out a way to see if their frequency counts are equal in number, but that wouldn’t necessarily tell me if my raters were counting the same incidents.  One method in traditional IRR calculation is to divide the period into intervals and simply mark if the behavior did or didn’t occur in an interval, but StudioCode allows us to be so much more accurate than using general intervals.  Also, I would be able to code leading events to the aggression to look for patterns in the function of the behavior (why the student was acting aggressively).  Any thoughts on some easy ways to report some of this data with scripting?

Some Thoughts As I Work: Questions, Comments, and Solutions

As I was working on Interrater Reliability calculations and scripting, I had some thoughts I would like to share.

Scripting

In this screencast I demonstrate how I used scripting to calculate 4 values that I will be able to graph for the single case research design part of my study. Using scripts is extremely helpful because each time I bring up a new coded video, it will automatically give me those values for that video. I will not need to calculate any numbers by hand. I only briefly point out my scripts, but if there are any questions, I can certainly explain the scripting language I used. 🙂 There is a pattern to scripting and StudioCode has some example scripts you can pull from and just alter to meet your needs. It’s definitely not as complicated as most computer programming languages. I didn’t really know anything about computer programming before and I was able to pick up scripting very quickly. There are plenty of tricks and shortcuts that Ryan helped me with, so if this is something you are trying for the first time, I would definitely run your ideas past one of the trainers.

Stacking Videos

Ryan and I did another training today and he helped me solve all of my immediate setbacks.  We discussed cameras and filming angles, stacking timelines, scripting, and Interrater Reliability.  I don’t want to overload any posts, so I will spread out what I’ve learned over several posts over the next few days.

Today’s post will be on multiple camera angles and stacking videos into one timeline.  

In my last post I explained my dilemma about cameras.  In educational research, there is a lot to consider when it comes to video including authenticity of situations, logistics of setting up/taking down cameras each day, and the privacy of students.  Ryan and I spent some time examining different camera options online and brainstormed some pros and cons.  We looked at the GoPro cameras because they were recommended to me.  They are fairly inexpensive and have a wide angle lens, but they are still larger than the cameras I already have and I would still have to find appropriate tripods or mounts for it.  We also looked at various “spy” type cameras that students could clip to glasses or a headband.   Most of these cameras would not film with the same ratio as my other cameras and they also would require format conversion.  While these aren’t huge problems, I think forcing 6-7 year olds to wear a camera on fake glasses or on a headband would threaten the authenticity of my results.  I can see them as a major distraction.  We looked into the Reflector app more too (since I will need a clear view of the iPad), but we could not figure out a way to stream just the video from the iPad.   I absolutely must keep the sound on the iPad because audio is a main feature of the eWorkbook.  Apparently at this point there is only a way to stream just video from a computer to an Apple TV, not from an iDevice.

Basically, we landed back where we started… we decided to work with what I have and just work on camera angles.  Rather than purchasing new cameras, I will only purchase tripods and tripod adapters for iPods.  I have 3 iPod Touches and 1 Kodak Easy Share camera that I will use to get three “overhead” shots of the three participants and one general classroom “standard” shot that includes all three students.  The good things about these cameras are that they already film in the correct format, they can film in the same aspect ratios, and they are very easy to import.  I will definitely have to spend some time in the classroom prior to starting research to find the correct angles though.  For my coding, the standard shot will be stacked with the overhead shot for each student so we can see the iPad screen, how the student interacts with it, and what is going on in the entire classroom.  When you watch the screencast, you will see the angles we came up with.  I am pleased with these views, but I also welcome any feedback on camera angles.

A Stacked timeline allows me to look at multiple angles at the same time and code onto one single timeline.  For anyone looking for quick directions on creating a stacked timeline, here they are:

Creating a Stacked Timeline:

– Create 2 separate movie packages for your videos

– Line up the play-heads to the starting cue on each video(so they will stack correctly)

– Move your “main” angle up to the front timeline (you can always rearrange later)

– File; Stack Timeline movies; Save as any name

– Options: Make sure “cut start time using – playheads” and “cut end time using shortest movie” are selected

– NOTE: This will only be a reference file

File; Save as a “Standalone package” (Can use same name if you save it to a different location first)

Delete other files to save space: 2 separate movie packages and the original stacked reference file

Video Angles

I haven’t had as much time to play around with these videos as I would have liked in the past bunch of days, but I did spend a few hours last night taking some sample videos of myself acting as a student.  I tried to mimic what I saw students doing in our pilot study so I could practice different angles.  I am having a really difficult time getting a good angle so I can see the student’s actions, see the iPad clearly, and know what is going on in the rest of the class.  I will definitely need at least two angles, which could be troublesome in terms of logistics, especially since we plan on having three student participants.  I don’t want the classroom to be full of cameras, and I don’t want the student to be distracted by the presence of a camera so close to them.

I could do an over the shoulder view of the iPad so I can see the student’s hands and the way they interact with the iPad:

IMG_1101

The only issue with this angle is that I cannot see the student’s face or body language to see if he or she is distracted.  If the student doesn’t manipulate the iPad for a while, at this angle it would be difficult to tell whether it is because he or she is distracted, or just reading.

I could also do an across the room view like the one below where I can see the entire body of the student.  This is important for the reasons described above.  This view unfortunately doesn’t show me what the student is doing on the iPad, but if I stacked both video angles, I could get a good picture of the student overall.  (I just don’t know what challenges coding stacked videos will bring).

IMG_1099

When conducting research in a classroom, especially at the elementary level, you need to consider the authenticity of the environment.  I am worried about needing multiple camera angles for multiple students because logistically I would need enough cameras, and I don’t want those cameras to be a distraction.  I was trying to find a way not to have so many cameras set up in the room so I looked for a way to see what the student was doing on the iPad without using a video camera.  I found a Mac app called Reflector that might work:

Each angle has its limitations and ideally, I could use all three.  I have a major concern about the sound with the reflector app, but if I could figure out a way to change the sound streaming option, it would be an excellent angle.  I guess this post is just a way for me to think out loud.  If anyone has a suggestion for me, I’d love to hear it.  Thank you to everyone who has provided me with feedback throughout this process…it has been extremely helpful in shaping my understanding of this research and the StudioCode software.