Home > conferences > Program Committee Meetings Considered Harmful

Program Committee Meetings Considered Harmful

The organizers of CSCW 2012 have started an intriguing experiment this year: a review process with an extra revise and resubmit phase. The goal is to try to find reasons to accept papers, rather than find reasons to reject them. Over all, I would say the experiment is a huge success. The quality of papers is just as good, and a lot of good work got saved.  Though it does have some unpredictable consequences, for example how will promotion and tenure committees view the conference now that the total acceptance rate is higher?  The quality of the work is just as good, but does everyone know that?

With the two-round review process, in fact most decisions on papers were already made before the face-to-face program committee (PC) meeting, which I’m at right now. We had only 27 papers to discuss here, of  388 submissions.  This raises the question: do we even need to hold a face-to-face PC meeting?  It costs a lot of money and a lot of carbon to bring everyone here.  If it’s not necessary, we shouldn’t have it for pragmatic reasons.  I want to argue, though, that the reasons to not have the PC meeting are more than pragmatic: it will actually improve the quality of the conference.

Before the conference, some number of reviewers and associate chairs (ACs) read the paper carefully and give their reviews.  Then they can discuss the paper via a discussion board for that paper, which retains the anonymity of the reviewers to one another.  At the PC meeting, a room full of associate chairs meet to discuss the paper. And here’s where something odd happens: people who haven’t read the paper offer their opinions.  So for example, yesterday one presenter said “this paper has interesting qualitative findings, but is somewhat under-theorized.  It’s about an interesting user population, but is mainly just descriptive.”  And then a long discussion ensued about whether to accept this kind of paper.  But most of the people in the discussion hadn’t read the paper. To me the discussion should turn on the quality of the actual paper.  I don’t think we can answer this question in the abstract. Giving serious weight to comments by people who haven’t read the paper is bizarre. I believe this leads to poorer quality decisions.

So what do you do if reviewers can’t reach agreement on a paper? I suspect that many of these cases can be resolved by adding an additional reviewer, and continuing to discuss it online. A synchronous conference call could possibly be arranged where needed. But we would make better decisions if ultimately the people participating in the conversation all had read the paper.  One downside of this approach is that PC meetings serve to calibrate expectations for how high the bar is.  But I think there are other creative approaches to helping people calibrate, including providing reviewers with a visualization.  This could include making visible how harsh or generous each reviewer is on average, compared to other reviewers of the same papers.

PC meetings do serve an important function for community building, helping junior peers become more central members of the community, and reflecting on where the field is going as a whole. These functions could be filled with a special meeting and dinner for ACs at the conference event which helps plan for future years.

I genuinely enjoy face-to-face PC meetings. I get to see such terrific people at them. But I think it’s time we question whether they are good idea. We may make better quality decisions without them.

Advertisements
Categories: conferences
  1. October 14, 2011 at 1:25 pm

    My policy as a program chair has always been to require two — preferably three — people AT THE PROGRAM COMMITTEE MEETING have read the paper under discussion.

    This is demanding on the part of the associate chairs, but that’s only right.

    I also prefer to have the committee discuss, at minimum, every paper that could possibly be accepted. The first duty of a scientific program committee is to avoid publishing work that is wrong; sometimes, it takes many eyes to uncover a faulty assumption or a flawed experiment.

    I believe program committee meetings are indispensable for teaching people how to review well. I know that my own sense of rigor — especially with respect to the work of senior researchers and prestigious labs — was honed by working on program committees with people like Frank Halasz, Cathy Marshall, Mark Frisse, Polle Zellweger, and Wendy Hall. Without that experience — the constraints of a phone call or email can’t replace it — I know that I’d have been a far worse reviewer and a useless program chair.

  2. October 14, 2011 at 1:43 pm

    At a recent PC meeting (not this one) as an AC – my first time on the PC for this conference, the room objected to me throwing out a paper that had good scores in favor of a paper that had borderline scores. I saw things moving in this direction and just pointed out to the PC “clearly you didn’t look at those reviews that gave it a good score – they were rather crummy and short – more so the article itself is lacking…the borderline paper is full of interesting controversy which is evident in its reviews and the article.” When my short defense was done, the PC stopped for a moment and agreed with me. Perhaps, rather problematically, telling people to “listen up bc clearly you all missed something by not reading” is not how one builds friends and centrality in the community – lucky for me this PC already knew me and already had formed their opinion. 🙂

  3. Daniel Avrahami
    October 14, 2011 at 7:58 pm

    Two quick comments Amy:
    First, I want to repeat what Bob had said today: The addition of the Revise and Resubmit aspect of the process is orthogonal from increased acceptance rate. That is, having an R&R process does not increase the acceptance rate; it is the number of papers that the PC chairs assign into the R&R category that influences the acceptance rate.
    Second, that at least we only discussed and potentially harmed 7% of the papers. So maybe harmful, but not catastrophic.
    – Daniel

  4. October 14, 2011 at 10:35 pm

    On the pragmatic front: a) 222,078 passenger miles travelled (48 people); b) CO2 emissions of 53.4 metric tons; c) 192,087 kWh expended over 12.5 hours of meeting time.

    49% of the emissions were due to the 11 participants who flew from farther than 6000 miles.

    There is no way a multi-site meeting tied together with video and audio would exceed the energy consumption/carbon footprint of this ftf meeting (even accounting for the embodied energy in the devices and infrastructure).

    A metric ton of gas is a volume the size of the Washington monument.

    Per capita annual CO2 emissions for the US are 17.5 metric tons (2008 data).

    • October 15, 2011 at 10:23 am

      In addition to the carbon, there are the dollars spent. I charge PC meeting travel to NSF grants. Which is within the rules as I understand them, but feels like stretching things a bit.

  5. Mor
    October 17, 2011 at 11:22 pm

    “And then a long discussion ensued about whether to accept *this kind of paper*” (emphasis mine).

    Somewhat echoing Mark — do the PC meetings play a role in establishing/persisting community norms, and informing/teaching new members of the community?

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: