Methodological Pitfalls in CSCW Research
CSCW research is exploding in popularity, and with good reason. It’s hard to think of an area of life that is untouched by collaborative uses of the Internet. But as I’ve been watching the growing body of research over the last few years, I’ve noticed some methodological challenges that pop up repeatedly. I want to highlight three problematic patterns:
1. Working with the Data We Have
So you start off with a set of research questions that meet some heuristic for Interestingness, and pick a real, deployed system for which those questions are relevant. Then you collect the data you can access, with available staffing and with limits on data access. The data you can get without herculean effort doesn’t quite answer the questions you started with. So you weave a story around the data you have. Which ends up lamely dancing around its inability to answer the Interesting questions, or addresses different convenience questions that aren’t especially interesting.
2. Over-Generalizing from a Toy System
To avoid Pitfall 1, you decide to design an entire system so you can control every aspect of it and collect richer data. You make a simple system so you’re not spending years developing, and anyway the point is in the user data–you’re not trying to make a deployable system. You pay or cajole people into trying out the system in a lab experiment or short-term deployment mode. Then you write up the results and attempt to generalize to problems of broader interest. But does what people do with a toy system really speak to what they do in more realistic situations? Probably not, but you make sweeping claims anyway.
3. Build a Giant System and Run out of Time and Resources
OK, so to avoid the toy system pitfall, you decide to build a real system. Something that real people will want to use. You spend two years doing software development. Then you spend another year in iterative design with initial testers. Then your money is spent and your students have graduated and… I know there was a reason we built this monster, but does anyone remember why? The study that the system was built to facilitate never gets properly done because we’re out of resources and all of this took so long that the world has moved on anyway.
These are pessimistic scenarios, but I’ve personally fallen into all three pits to one degree or another. Are these familiar to you too? I’d like to start a community discussion about what we as a field can do to avoid them. My first hunch is that maybe we need larger scale projects. All of these are to some degree side effects of limited resources. Maybe one or two PhD students working with one faculty member isn’t the right size intervention to accomplish real work in this field. That’s one idea. What else?