Home > Uncategorized > An Epidemic of Pointless Social Computing Research

An Epidemic of Pointless Social Computing Research

I went to a talk a while back where a senior researcher analyzed some large-scale social computing data, and proved that it displayed an elegant mathematical property.  I raised my hand during the question period and asked (as politely as I could muster), “Why does this matter?  Does this have some kind of broader implication or application?”  The researcher had no answer for me.  In fact, he seemed puzzled by the question.

I’ve been going to a lot of talks like that lately–they seem to be breeding.  People are playing with big data and coming up with incredibly clever results–with no evident broader implications.  Lately, Twitter data seems to be a chief culprit.  It’s so easy to get (or it was), and look at all the cool analysis you can do on it!  I’m betting most of the those papers will rot uncited. I hope when our enthusiasm for this new big data toy wears off, people will invest their energies in results that matter.  Of course, defining “matter” is the challenge.

What is the distinction between basic research and playing with data as a clever puzzle game?  How do you even tell the difference?  That’s the hard part. Personally, I’d like to see more people doing user-centered design: starting with problem statements that are significant for some group of people for some reason.  

Categories: Uncategorized
  1. lanayarosh
    April 23, 2013 at 11:37 am

    I agree with your point, but it’s also hard to know ahead of time what level of broader application would be useful. Is it broad enough if I connect my results to other research on micro-blogging? Do I have to connect it to even broader social networking research?

    I had this issue with my CHI paper… I tried to draw broader implications for social computing, but actually the reviewers wanted something much more specific to the particular context I was studying. In the middle, there was a good compromise, but it’s hard to know where that line is ahead of time.

  2. April 23, 2013 at 11:55 am

    Although I agree with your point that we shouldn’t use big data to find and present useless statistical properties of data, I think that your post could benefit from a more careful analysis of what type of quantitative (big data) practices are, in fact, useful.

    For example, some quantitative results may inspire interaction design, e.g. Dunbar’s number which shows the typical size of users’ group of friends. Similarly, quantitative results may be used to provide scientific “proof” (or evidence) for the effectiveness of newly developed designs.

    It is easy to dismiss big data as a “numbers game”. Let’s instead make optimal use of the exciting new opportunities to inform and validate design!

  3. April 23, 2013 at 12:08 pm

    Is this really an epidemic with social computing data mining research, or is it a problem with __bad__ social computing data mining research? Law of large numbers says there will be a lot of mediocre and bad papers, and it’s easy to focus our attention on them.

    This plays out in predictably critique-able ways:
    – Bad data mining research has few generalizations to theory, design, or future data mining models
    – Bad HCI systems/applications research looks like advanced development rather than research
    – Bad algorithms research shaves insignificant terms off of asymptotic bounds
    – Bad CS systems research feels like tiny optimizations without a bigger idea
    – and so on.

    The one that really got me was when an AI researcher presented a bad project and excused it by saying, “Well, this isn’t really AI research — it’s more HCI research.” No, sir, in my opinion it wouldn’t be considered HCI research either.

    • April 23, 2013 at 2:02 pm

      Michael, I agree that part of the problem is generic bad research, which is an equal opportunity problem.🙂
      But I also think there is a particular genre of bad research which is happening at this historical moment
      more than in the past.

  4. anonymous
    April 23, 2013 at 2:37 pm

    impact is not necessarily linked to application. In that case what would one say about pure math research or theoretical physics research? The fact that they don’t immediately relate to real world applications doesn’t make them less scientific or impactful.

    • April 23, 2013 at 3:36 pm

      Totally agree. But I think you’ll also agree that there also is some work that is pointless. I don’t know how you tell the difference! We could ground this in a debate about specific papers, but let’s not go there.😉

  5. April 23, 2013 at 11:02 pm

    I define bad research as research that has meaningless evaluation. A lot of research has this problem. I would actually estimate that the vast majority of the research I’ve read in my post-graduate career has the property of meaningless evaluation.

    Good research is difficult and very, very rare. I look for whether the best papers in a field are truly useful and discount the rest as people trying to find a way to make a living.

  6. Mark Handel
    April 26, 2013 at 4:39 pm

    From my experience, reviewers seem very skittish about social computing work that falls outside of mainstream experiences. And, at least for me, it the smaller, more “marginal” communities where it is easier to gather enough data and understand what’s going in both qualitative and quantitative aspects. But then, that is often at the edge of people’s comfort zones.

    I’m probably overly sensitive to these issues right now; I have two nice data sets, one of which I can’t publish (proprietary information) and one of which gets rejected (uncomfortable topic area).

  7. April 27, 2013 at 11:30 am

    I challenge Bart K’s assumption that Dunbar’s number of 150 is useful. t assumes a simple society, but we live in multiple complex networks. Current research suggests an N of 600+. See my “Is Dunbar’s Number Up?” in British J of Psychology. http://homes.chass.utoronto.ca/~wellman/publications/dunbar/dunbar%20critique%20bjp.pdf

  8. April 27, 2013 at 12:31 pm

    Well Barry, it was at least useful enough to inspire your subsequent work on the topic, right?😉

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: