Archive for May, 2011

Should you believe Wikipedia?

May 18, 2011 15 comments

I got a nice email today from librarian Tedd Guedel of Herzing University asking about the reliability of Wikipedia (he saw the announcement for a talk I gave last October, ““How Wikipedia Really Works, and What This Means for the Nature of “Truth”.”) With Tedd’s permission, I quote:

“As a rule, I steer students away from Wikipedia as a valid academic source. Do you have a power point or any information that you gave during your lecture that you could share with me? … Our old IT director Loved Wikipedia because “it is maintained by the masses” I do not like Wikipedia because “it is maintained by the masses.”

So who’s right, Tedd or his IT director? Luckily in this case I don’t have to take sides–they both are right. The answer is: what page on Wikipedia? How many people have edited it? How many people are ‘watching’ it? I will argue that a popular, high profile Wikipedia page is the most accurate reference that has ever been created in the history of the written word. (Really!) A low-profile page that few people have edited is unreliable. It all depends on how many people have checked the article and its references.

To explain why, it might help to discuss how a refereed journal article is reviewed. Articles in high-quality, peer reviewed journals are generally considered the gold standard for reliability. An author submits an article to a journal, and it is sent out to approximately three experts in the field for review. Each expert reads the paper carefully, and sends detailed comments. The experts are anonymous to the author, and can make critical comments without fear of giving offense. The author usually revises the article, and the experts read it again. Most articles are rejected. Nothing is published until the experts are happy. The reliability of the article comes from the number of people who have reviewed it, the special expertise of those people, and how carefully they reviewed it. Once the article is printed, it can not be updated. Corrections have to happen in a follow-on article, which may not happen and may not be evident when you are looking at the original. Exact customs vary by field, but this is the general pattern.

What happens when a popular Wikipedia article is created? The birth of a Wikipedia article on a high-profile topic is a beautiful thing to witness. For example, Brian Keegan notes that in the 100 hours after the Sendai earthquake and tsunami in Japan, 1,727 people made 6,931 edits to 49 relevant articles. The main Sendai quake page at the time of this writing has 289 references. Everything about it has been checked and rechecked. Today 349 people have the article on their “watchlist”–the list of pages they monitor for changes. (Not everyone actually checks their watchlist of course.) Vandalism on Wikipedia is typically removed quickly–Fernanda Viegas and Martin Wattenberg found that it is often corrected in seconds.

Next time someone is nominated to the US Supreme Court or becomes the next Pope, watch their page as it evolves.  On any Wikipedia page, you can click the “View History” tab and see all the edits to the article over time.  Over the period of a few days, the newly famous person’s page evolves from a few sentences to a complete concordance on their life and work–with every fact supported by references, and anything unsupported removed quickly. It’s astonishing to witness.

So what would you rather have–something checked by three experts over six months to a year, or something checked by 1,727 people in the first 100 hours?  And remember that many of those 1,727 people are checking references and not allowing anything that isn’t documented.  Also remember that the refereed journal article is fixed at a moment in time, and beyond that any errors or new developments aren’t included. A Wikipedia article is updated continuously. Of course the purpose of a journal article and encyclopedia article are entirely different–one presents new knowledge and the other summarizes consensus and explicitly forbids original research. They’re not comparable. But if you believe that reliability of knowledge is in relation to how many people check it and how carefully, then a popular Wikipedia article does pretty well.  Amazingly well, in fact.

Particularly surprising to me is the fact that topics on controversial issues can be quite good. For example, I would have guessed that the article on whether vaccines cause autism would be a cesspool of controversy and misinformation. But it’s not. It reviews the history of the controversy in comprehensive detail (supported by references) and unequivocally says that the original paper suggesting a connection has been proven a fraud and there is a scientific consensus that there is no such link.  Hooray!

But those are all examples of high-profile articles. What about low profile ones?  Click the ‘random article’ button on the left hand column of the Wikimedia software, and see what you get.  Often you’ll get something that’s barely been started–a “stub” in Wikipedia parlance.  It’s possible to put something unsupported in an obscure article, and it may not be checked. Famously, a prankster wrote that journalist John Seigenthaler was a suspect in the assassination of John F Kennedy.  The error remained there for over six months until a friend of Seigenthaler’s noticed it.  I should note that this happened in 2005 and the culture of Wikipedia has changed since then–things are now checked more carefully.  But is it possible for a prank or honest error to linger? If it’s in an article that is not high profile, absolutely.

Another  problem is circular references. It happens like this: a journalist uses Wikipedia as a source for some information, and publishes an article without having any other  source. Later, a kind Wikipedia editor notices that the article is unsupported and searches for good references–and finds the journalist’s article and cites it!  (Oops…)

It’s not surprising that people are confused about whether to believe Wikipedia–the truth is complicated. I believe today we have a crisis in epistemology–no one knows what to believe any more. But it’s also a teachable moment. A moment to teach students about peer review and the importance of references and how to think critically about the reliability of everything they read.

Gender Swapping Reinforces Stereotypes

May 17, 2011 1 comment

At CHI 2011 in Vancouver, Nick Yee and colleagues presented a fun note about gender swapping on World of Warcraft: “Do Men Heal More When in Drag?
Conflicting Identity Cues Between User and Avatar
.” There are many gender-based stereotypes about people’s behavior on MMOs. For example, people assume that women are more likely to play healing characters. However, Yee found that women are in fact not more likely to heal than men. However, men playing female primary characters are more likely to heal a lot.  When gender swapping, players live up to their stereotype of gendered behavior, even if that stereotype is not true.

This fascinates me, particularly because Josh Berman and I found the same thing in our study of an online identity game we created in 1999, The Turing Game.  In the Turing Game, a panel of players pretends to be a particular identity–for example women. (Game types were user created, so people played lots of fun games like who is from Canada, who is under 30, who is a parent, and more.)  The audience asks questions, and votes on who they think is telling the truth. After the game, contestants reveal their real identities and discuss how everyone knew the truth or was fooled. In many of these post-game conversations, audience members would say things like, “I knew you were really a woman, because you use long sentences with lots of dependent clauses. Women talk a lot. Men say things like ‘I’ll be back.'”  The only problem with this is that Susan Herring can conclusively show you that men use more words per conversational turn online.  The stereotype is wrong. And just as Yee found 12 years later, gender swapping  reinforces stereotypes.

We are currently redesigning and reimplementing The Turing Game as a web game with Facebook Connect. Our challenge in the redesign is to figure out how to help these playful explorations yield deeper insights and less bandying about stereotypes (whether true or false).

%d bloggers like this: