Home > Uncategorized > Methodological Pitfalls in CSCW Research

Methodological Pitfalls in CSCW Research

CSCW research is exploding in popularity, and with good reason. It’s hard to think of an area of life that is untouched by collaborative uses of the Internet.  But as I’ve been watching the growing body of research over the last few years, I’ve noticed some methodological challenges that pop up repeatedly.  I want to highlight three problematic patterns:

1. Working with the Data We Have
So you start off with a set of research questions that meet some heuristic for Interestingness, and pick a real, deployed system for which those questions are relevant.  Then you collect the data you can access, with available staffing and with limits on data access.  The data you can get without herculean effort doesn’t quite answer the questions you started with. So you weave a story around the data you have.  Which ends up lamely dancing around its inability to answer the Interesting questions, or addresses different convenience questions that aren’t especially interesting.

2. Over-Generalizing from a Toy System
To avoid Pitfall 1, you decide to design an entire system so you can control every aspect of it and collect richer data.  You make a simple system so you’re not spending years developing, and anyway the point is in the user data–you’re not trying to make a deployable system. You pay or cajole people into trying out the system in a lab experiment or short-term deployment mode.  Then you write up the results and attempt to generalize to problems of broader interest.  But does what people do with a toy system really speak to what they do in more realistic situations?  Probably not, but you make sweeping claims anyway.

3. Build a Giant System and Run out of Time and Resources
OK, so to avoid the toy system pitfall, you decide to build a real system. Something that real people will want to use.  You spend two years doing software development. Then you spend another year in iterative design with initial testers. Then your money is spent and your students have graduated and… I know there was a reason we built this monster, but does anyone remember why?  The study that the system was built to facilitate never gets properly done because we’re out of resources and all of this took so long that the world has moved on anyway.

These are pessimistic scenarios, but I’ve personally fallen into all three pits to one degree or another.  Are these familiar to you too? I’d like to start a community discussion about what we as a field can do to avoid them. My first hunch is that maybe we need larger scale projects.  All of these are to some degree side effects of limited resources. Maybe one or two PhD students working with one faculty member isn’t the right size intervention to accomplish real work in this field.  That’s one idea.  What else?

Categories: Uncategorized
  1. May 30, 2012 at 12:14 pm


    Thanks for sharing these thoughts. In my short time as a grad student thus far, I have already seen these pitfalls in research over and over again. I feel like very often students don’t put research into perspective, often times trying to overcelebrate the little victories without understanding that those successes are indeed valuable, but more so as the first step on the path to a greater change. It seems like the community at large needs to start valuing research that takes on these larger challenges and encourages students to do so as well as faculty. It would be great to spend time at the next town hall (or even PC meetings, etc) and map out what some of the “grand challenges” within the CSCW space are. Where are these big areas that we can come and work around, and then how can we do so to spur intervention, rather than just pat ourselves on the back.

    That being said, I think celebrating progress is great, but it seems like people do research very often to say they’ve done the research. Perhaps that is my perspective as a newcomer to the field, but I’d like to see more people talking about how this research has a tangible impact and how they are moving from where they are to making that impact.

    On a separate, but related note, I think talking about our research outside the community is also valuable. There are an incredible number of skills and talented people in the CSCW community, and it would be great to open up our research to a broader audience to understand what problems they think we could help tackle as well, especially those that we may be blind to.

    Thanks again for sharing.


  2. May 30, 2012 at 12:15 pm

    Partnering with industry and using their data sets is a way to avoid Pitfall 1.

    Making careful conclusions and describing limitations can help avoid Pitfall 2.

    To avoid Pitfall 3, we might learn something from robotics research (maybe this was your suggestion?). As I understand it, everyone in a robotics lab works on the same robot. They are way too complicated and expensive to have one for each student. So perhaps some CSCW researchers should orient their labs around building and maintaining a single “giant system” that every student is expected to work on.

    • May 30, 2012 at 12:20 pm

      Agreed Kurt–call that the Minnesota approach (MovieLens, Cyclopath)? Which has its own challenges….

  3. May 30, 2012 at 12:23 pm

    Collaborating with industry mitigates many of these problems, if you can persuade a researcher at Facebook or Google or Microsoft that your Interesting questions are truly Interesting. Then you get data from a massive system with real users and much better infrastructure for queries and analysis. Often the kinds of fundamental questions CSCW researchers want to understand are the ones industry researchers are too busy to delve deeply into, so it’s a win for both sides. The trick is to frame your research in a way that informs product design or vision, so it’s useful to the company, too, and worth the cycles of the researchers and engineers on the inside who help. But the best CSCW research should contribute to both theory and practice, anyway.

    • May 30, 2012 at 12:29 pm

      Moira, I couldn’t agree more. And did I mention that you’re my new best friend? 😉

      One challenge for academia-industry collaborations is whether it is possible to end up with research that is unflattering to or does not mesh with the agenda of the industrial partner. Which is the same problem that comes up in working with other kinds of partners like nonprofits. (“Hmmn, I want to keep working with these folks, so….”) The power relations get complicated.

      • May 30, 2012 at 1:02 pm

        Oddly, it worries me that research now *requires* an industry partner. I’d rather researchers just fess up and disclose what they have in their dataset and the limitations of what they suggest…you can improperly report on data with 10k posts as you can 100M posts.

      • May 30, 2012 at 2:36 pm

        Ayman, the people who think industry partnerships are required for social computing research are often the same people who think “big data” analysis is required.

      • June 2, 2012 at 8:55 pm

        One concern I have for industrial-academic research is that of reproducibility of results. This is an issue for other setups too – it’s probably hard to reproduce some MovieLens results without your own MovieLens – but what should the standards for reproducibility be? How widely-accessible should the data & capabilities necessary to reproduce research be?

  4. beki70
    May 30, 2012 at 12:54 pm

    Another challenge of the large systems approach is making sure there’s enough there there for everyone who needs a there to be there… 🙂

  5. May 30, 2012 at 3:02 pm

    Hi Amy,

    This is a great discussion. Any project involving technology development (software or hardware) has lots of challenges and when you have to research on them, another set of challenges will appear. I agree with Kurt and Moira. We as researchers also need to have a set of good values when presenting the results of our research and not generalize what can not be generalized.
    It would be great to have more resources at the universities, but given the actual global crisis this will not happen any time soon, so we need other solutions for this sort of problem, perhaps just being more modest and honest about the results?

  6. June 2, 2012 at 9:02 pm

    One thing that I think might be a promising line of thought for system-building is to adapt Minimum Viable Product thinking from the startup world. What is the smallest/simplest thing I can build that will let me answer the questions I have (and provide sufficient value to users that they will want to try it)?

    I don’t know if this is the mindset that led to it, but Jilin Chen’s 0088 stuff seems to fit this mold. A simple, minimal tool that provided concrete value to users and facilitated the study.

    Now, this doesn’t work particularly well for studying usage of native features in existing systems, but it seems to me that there is a lot of space between toy systems and large systems that is profitable for promoting research.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: