Program Committee Meetings Considered Harmful
The organizers of CSCW 2012 have started an intriguing experiment this year: a review process with an extra revise and resubmit phase. The goal is to try to find reasons to accept papers, rather than find reasons to reject them. Over all, I would say the experiment is a huge success. The quality of papers is just as good, and a lot of good work got saved. Though it does have some unpredictable consequences, for example how will promotion and tenure committees view the conference now that the total acceptance rate is higher? The quality of the work is just as good, but does everyone know that?
With the two-round review process, in fact most decisions on papers were already made before the face-to-face program committee (PC) meeting, which I’m at right now. We had only 27 papers to discuss here, of 388 submissions. This raises the question: do we even need to hold a face-to-face PC meeting? It costs a lot of money and a lot of carbon to bring everyone here. If it’s not necessary, we shouldn’t have it for pragmatic reasons. I want to argue, though, that the reasons to not have the PC meeting are more than pragmatic: it will actually improve the quality of the conference.
Before the conference, some number of reviewers and associate chairs (ACs) read the paper carefully and give their reviews. Then they can discuss the paper via a discussion board for that paper, which retains the anonymity of the reviewers to one another. At the PC meeting, a room full of associate chairs meet to discuss the paper. And here’s where something odd happens: people who haven’t read the paper offer their opinions. So for example, yesterday one presenter said “this paper has interesting qualitative findings, but is somewhat under-theorized. It’s about an interesting user population, but is mainly just descriptive.” And then a long discussion ensued about whether to accept this kind of paper. But most of the people in the discussion hadn’t read the paper. To me the discussion should turn on the quality of the actual paper. I don’t think we can answer this question in the abstract. Giving serious weight to comments by people who haven’t read the paper is bizarre. I believe this leads to poorer quality decisions.
So what do you do if reviewers can’t reach agreement on a paper? I suspect that many of these cases can be resolved by adding an additional reviewer, and continuing to discuss it online. A synchronous conference call could possibly be arranged where needed. But we would make better decisions if ultimately the people participating in the conversation all had read the paper. One downside of this approach is that PC meetings serve to calibrate expectations for how high the bar is. But I think there are other creative approaches to helping people calibrate, including providing reviewers with a visualization. This could include making visible how harsh or generous each reviewer is on average, compared to other reviewers of the same papers.
PC meetings do serve an important function for community building, helping junior peers become more central members of the community, and reflecting on where the field is going as a whole. These functions could be filled with a special meeting and dinner for ACs at the conference event which helps plan for future years.
I genuinely enjoy face-to-face PC meetings. I get to see such terrific people at them. But I think it’s time we question whether they are good idea. We may make better quality decisions without them.