Archive for the ‘research ethics’ Category

Talking Sense about MOOCs and Online Education: the New Post-Hype Era

August 1, 2014 Leave a comment

At this year’s CRA Snowbird conference (the every-other-year gathering of chairs of CS departments), I organized two panels on MOOCs and online education. While I’m told that Snowbird 2012 was dominated by hyperbole about MOOCs, our discussion this year was eminently sensible. In our panel “MOOCs and Online Education: The Evolving Big Picture,” Nelson Baker (Georgia Tech), John Mitchell (Stanford), and Marian Petre (The Open University) talked about the realities. There’s a lot we can do with online education. It’s wonderful that GT’s new online master’s of computer science is reaching working professionals who otherwise couldn’t pursue higher education. But how to do it well and how to make the bottom line add up are challenges. It’s not cheaper if you do it right. We had standing-room only for the panel, and both the positive hype and negative hype were absent. People were talking sense.

In our second panel, we discussed MOOCs and online education as active areas of computer science research. Marti Hearst (UC Berkeley), Scott Klemmer (UCSD), and Rob Miller (MIT) showed some current research in progress on how to design new software for online education inspired by good pedagogy. Right now we’re still in the horseless carriage stage of online ed–trying to understand the new medium in terms of the old one. How to do this well is an open area for research. And we need research done in both ischools, ed schools, and computer science departments. There is a complicated interaction between what the technology can do and what good pedagogy says we should do. Making those work together is a challenge. And department chairs and deans need to think hard about whether they are able to fully support faculty members doing such interdisciplinary work.

One thorny area that needs further community discussion is research ethics. Whenever you do research on students, you need to recognize that there is an unavoidable power relationship between faculty and students, particularly if investigators are doing research on their own classes. Petre emphasized that as faculty we have a duty of care. The rule book on the ethics of researching real students in online classes is still being written, and it has more nuance and complication than recent controversies about social network sites conducting research on their members.

What was noticeably absent from our online ed mini-track was hype–both positive and negative. The truth is somewhere in the middle, and is much more complicated than you might think. And we’re just at the beginning.

I had a funny conversation years ago with a faculty member at MIT who has taught artificial intelligence (AI) for many years. At the time, AI was unfashionable. And he said he liked that better, because when AI was trendy they got lots of shallow people going into the field for the wrong reasons–just because it was “hot.” During the “AI Winter” when it was unfashionable, he had a smaller influx of potential students–but nicer ones, who were more sincerely interested in the discipline. Thank heavens we have gotten past the latest hype bubble about online ed, and are left with sincere people working on some interesting and worthy problems!


More on TOS–Clarifications

February 14, 2010 1 comment

Thanks for the interesting emails and Facebook comments about TOS (you all should post comments here!) A few clarifications:

ACM Code of Ethics and TOS

The ACM Ethics Code says:

1.5 Honor property rights including copyrights and patent.

Violation of copyrights, patents, trade secrets and the terms of license agreements is prohibited by law in most circumstances. Even when software is not so protected, such violations are contrary to professional behavior. Copies of software should be made only with proper authorization. Unauthorized duplication of materials must not be condoned.

To me, that adds up to saying you must respect Terms of Service. We could debate whether TOS and ‘terms of license agreements’ mean the same thing–one phrase is traditionally used for online services and the other for software you buy. But a TOS I think is a kind of software license agreement, and hence the above applies.  And the intent–to respect others’ intellectual property–applies too. Contents of an online service are generally considered intellectual property of the corporation. (We could have an even more complicated argument about that point.)

Additionally, section 2.8 of the ACM code  says you should access computer systems only when authorized to do so. You’re only authorzied to do so for an online service when you abide by the TOS, so again this reinforces the point.

In any case, I wanted to clarify that the ACM Code of Ethics doesn’t precisely mention TOS, and all of the above is open to interpretation

This isn’t about Facebook & I’m Not Criticizing Facebook

This isn’t just about Facebook–the TOS of many sites cause problems for researchers. Facebook is a key example because it’s so big that many people want to study it, and its terms are among the more restrictive.

I’m not in any way criticizing Facebook or its TOS. They are a corporation and have a fiduciary responsibility to protect the interests of the corporation. Their TOS are fairly well designed for that purpose.

One of the fascinating thing about the Facebook TOS in particular is that they are grounded in a system of selective enforcement. The TOS are not strictly and consistently enforced. Is this a matter of limited resources?  I’m sure that’s the biggest part of it. But it’s also a strategy for protecting themselves.  They can’t define clearly what sort of use of Facebook data is OK and what’s not, so they make it mostly all against TOS, but then choose what counts as a bad enough violation to enforce. It’s an incredibly complicated socio-technical system.

Size of the Community Counts

Lynda Mitchell, founder and President of the site (KFA), wrote to mention that their TOS say:

Announcements about research projects, solicitations of subjects for interviews or studies, conducting surveys, polls, or otherwise gathering information about members for anything other than personal use without the expressed permission of KFA.

That sounds extremely sensible to me. As a long-time member of KFA, I’d be furious if someone was researching my posts there without careful review. People post private medical info there–and also get help for problems with teachers at school, family members… lots of sensitive stuff. So how is KFA different from Facebook?  The sensitivity of the data is the first obvious point (though people do get pretty personal on Facebook sometimes too). But more important is the size. It’s much more invasive to study a small group compared to a big one. For example, in studying chatrooms, Jim Hudson and I did a study that found that for every 13 additional people present, we were half as likely to be booted out of the room for saying we were studying it. The bigger the group, the less people care.  KFA is an intimate group, and I’d want Lynda to think very carefully before allowing folks to even ask for study participation.

I still in the end think that we need to follow TOS for sites we study. Even on a big site like Facebook. But it’s a much more complicated question than it seems at first glance.

Terms of Service and Online Research–Inconsistency in Enforcment

February 13, 2010 1 comment

There was lots of talk about Internet research ethics at the CSCW conference, and the topic kept returning to Terms of Service (TOS). If you study an online site, do you need to abide by the site’s TOS?  For many sites, most notably Facebook, the TOS may make research near impossible. So does that mean the site can only be studied with the consent of the corporation? What if the corporation doesn’t feel like being studied? Sure, it’s their intellectual property. But if a site becomes an important part of culture and society, should the corporation have total control over how we try to understand it?

Is following TOS an ethical issue, or just a legal one? TOS are there to protect the interests the corporation. But they also often protect the interests of the users, because that’s usually in the best interests of the corporation. They tend to protect users from the actions of third parties (like outside researchers), but let the company itself do whatever it wants. For academic research, Institutional Review Boards (IRBs) and codes of research ethics are there to protect human subjects in studies. So at CSCW, several researchers argued that abiding by TOS is more of a legal question than an ethical one. The ethics are already covered by IRBs, and TOS are written with a business and not ethical mandate.

As a matter of law, it’s not clear that all TOS are legal. You can write anything you want in a TOS–that doesn’t mean it’ll stand up in court. But we won’t really know what will stand up in court until it’s actually litigated. And it’s rare for anyone to have the time or money to want to do that. Can a corporation prevent you from analyzing data posted openly online? Does it matter if it’s accessible without a password? Is violating such a policy a legal contract violation, or just grounds to lose your account?  We honestly don’t know.

The icing on the cake is that third parties outside of academic research don’t usually follow TOS. People at the conference were throwing around stats about what percent of Facebook applications violate the TOS. Most seem to. So the TOS aren’t an ethical document, it’s not clear if they’re truly legal, most people don’t really follow them… and yet academics are supposed to follow them meticulously?  Does this make sense to you?

If you argue that this is a legal issue and not an ethical one, there’s still  one problem: the ACM’s professional code of ethics explicitly says that we must abide by TOS.  When this was raised during an ethics panel discussion at CSCW, one senior researcher in the audience said then he’d resign his ACM membership if necessary. To further complicate things, other professional societies don’t have such rules. Many communities not only don’t recognize TOS, but also don’t believe that what they do is human subjects’ research at all.

At a recent major conference, a paper that clearly violated Facebook TOS was brought to the conference chair for discussion by the program committee. The chair said, “Print it!  They can sue us if they want.” But here’s the problem: at a different major conference the same year, multiple papers were rejected for not following the same TOS.  I’m not sure how much I care whether we follow TOS as a field, but I care very much that things are consistent and fair. One researcher forgoes a golden opportunity to do research because he feels he must abide by TOS, another goes ahead and does it. Among the ones who go ahead and do it, one gets a rejection, another gets published.  The situation is unfair. And it particularly annoys me that the net outcome is that the nice folks who decide to play by the rules get punished. As a field we need to get our story straight and find some way to have more consistent outcomes.

I’m a rule following sort of a person. I’m inclined to say, all things considered, that we should follow online TOS. And as a result we need new ways to work with corporations to get access to date without violating TOS.  At CSCW I met a grad student who is paid a salary of $1/year by Facebook, and that lets her get access to data. I wish more folks could get that deal–particularly if they are carefully screened and trained in how to handle the data safely.  But I have to admit the folks arguing that we don’t need to follow TOS have some surprisingly cogent points to make. One way or the other, it would be awful nice if we could make a decision and be consistent about it.

One possibility is for ACM conferences to require people submitting papers to explicitly state whether their work was reviewed by an IRB or other ethical board, and whether it abides by relevant TOS, and reject the work if it does not. This is an appealing idea. At least we’ll be consistent. The problem, though, is that we’ll simply be chasing this sort of work to other venues. Does that really help the state of affairs?  It’s not ideal, but I think it’d be an improvement.

%d bloggers like this: