Archive for October, 2011

Program Committee Meetings Considered Harmful

October 14, 2011 7 comments

The organizers of CSCW 2012 have started an intriguing experiment this year: a review process with an extra revise and resubmit phase. The goal is to try to find reasons to accept papers, rather than find reasons to reject them. Over all, I would say the experiment is a huge success. The quality of papers is just as good, and a lot of good work got saved.  Though it does have some unpredictable consequences, for example how will promotion and tenure committees view the conference now that the total acceptance rate is higher?  The quality of the work is just as good, but does everyone know that?

With the two-round review process, in fact most decisions on papers were already made before the face-to-face program committee (PC) meeting, which I’m at right now. We had only 27 papers to discuss here, of  388 submissions.  This raises the question: do we even need to hold a face-to-face PC meeting?  It costs a lot of money and a lot of carbon to bring everyone here.  If it’s not necessary, we shouldn’t have it for pragmatic reasons.  I want to argue, though, that the reasons to not have the PC meeting are more than pragmatic: it will actually improve the quality of the conference.

Before the conference, some number of reviewers and associate chairs (ACs) read the paper carefully and give their reviews.  Then they can discuss the paper via a discussion board for that paper, which retains the anonymity of the reviewers to one another.  At the PC meeting, a room full of associate chairs meet to discuss the paper. And here’s where something odd happens: people who haven’t read the paper offer their opinions.  So for example, yesterday one presenter said “this paper has interesting qualitative findings, but is somewhat under-theorized.  It’s about an interesting user population, but is mainly just descriptive.”  And then a long discussion ensued about whether to accept this kind of paper.  But most of the people in the discussion hadn’t read the paper. To me the discussion should turn on the quality of the actual paper.  I don’t think we can answer this question in the abstract. Giving serious weight to comments by people who haven’t read the paper is bizarre. I believe this leads to poorer quality decisions.

So what do you do if reviewers can’t reach agreement on a paper? I suspect that many of these cases can be resolved by adding an additional reviewer, and continuing to discuss it online. A synchronous conference call could possibly be arranged where needed. But we would make better decisions if ultimately the people participating in the conversation all had read the paper.  One downside of this approach is that PC meetings serve to calibrate expectations for how high the bar is.  But I think there are other creative approaches to helping people calibrate, including providing reviewers with a visualization.  This could include making visible how harsh or generous each reviewer is on average, compared to other reviewers of the same papers.

PC meetings do serve an important function for community building, helping junior peers become more central members of the community, and reflecting on where the field is going as a whole. These functions could be filled with a special meeting and dinner for ACs at the conference event which helps plan for future years.

I genuinely enjoy face-to-face PC meetings. I get to see such terrific people at them. But I think it’s time we question whether they are good idea. We may make better quality decisions without them.

Categories: conferences

Over Sharing

October 7, 2011 1 comment

I might play a quick game of Bejeweled Blitz right now, but my friends would see my score. I’m kinda bad at it. It shows your high score for the week to all your friends.  I like the game because it takes one minute to play, and a couple one-minute games is all I have time for most days. But it takes me five or six games at least to get a non-embarrassing score.  Mid-week, I’ll stop in and play one game. At the start of a new week, I know I need 15 to 20 minutes to achieve something approaching dignity in the score my friends see.  And lately I just haven’t had 15 minutes, so I’ve stopped playing altogether.  There’s no way to tell it “don’t share my score.”

More and more social computing apps are over-sharing.  Especially Facebook. On Facebook yesterday in the new  scrolling timeline feature that shows what your friends are doing, I saw a work colleague say something… something edgy to a male friend of his I don’t know.  He would never say anything like that at work. I wanted to cover my eyes–I didn’t need to see that.  I can’t un-see it.

If Facebook’s increased visibility causes problems for work/personal life boundaries, I can only imagine what issues it causes for people who are in a dating phase of life. I am having nightmarish visions of the drama that ensues with people watching their beloved post an innocent comment on a friend’s page and agonizing over whether there is flirting taking place.  This inadvertently happened to me in prehistory (the mid 1990s). Our sysadmin had accidentally left our UNIX history files (the record of commands you’ve typed) readable to others on the system by default, and a fellow grad student I’d gone on a few dates with jealously demanded an explanation for why I was fingering (looking up the online status of) my friend Brian so much.  Um, Brian and I were working on a class project together?

I don’t know why I care whether my friends see my Bejeweled score.  I don’t really–I don’t have any problem blogging about my Bejeweled ineptitude. But it’s just enough disincentive for me to stop playing that game. Going forwards, I imagine sites in general will land on different spots on the “how much to share” spectrum, and people will pick sites to use that are comfortable for their style. But as site designers we still need to understand more deeply how much sharing is too much.

%d bloggers like this: