I got a nice email today from librarian Tedd Guedel of Herzing University asking about the reliability of Wikipedia (he saw the announcement for a talk I gave last October, ““How Wikipedia Really Works, and What This Means for the Nature of “Truth”.”) With Tedd’s permission, I quote:
“As a rule, I steer students away from Wikipedia as a valid academic source. Do you have a power point or any information that you gave during your lecture that you could share with me? … Our old IT director Loved Wikipedia because “it is maintained by the masses” I do not like Wikipedia because “it is maintained by the masses.”
So who’s right, Tedd or his IT director? Luckily in this case I don’t have to take sides–they both are right. The answer is: what page on Wikipedia? How many people have edited it? How many people are ‘watching’ it? I will argue that a popular, high profile Wikipedia page is the most accurate reference that has ever been created in the history of the written word. (Really!) A low-profile page that few people have edited is unreliable. It all depends on how many people have checked the article and its references.
To explain why, it might help to discuss how a refereed journal article is reviewed. Articles in high-quality, peer reviewed journals are generally considered the gold standard for reliability. An author submits an article to a journal, and it is sent out to approximately three experts in the field for review. Each expert reads the paper carefully, and sends detailed comments. The experts are anonymous to the author, and can make critical comments without fear of giving offense. The author usually revises the article, and the experts read it again. Most articles are rejected. Nothing is published until the experts are happy. The reliability of the article comes from the number of people who have reviewed it, the special expertise of those people, and how carefully they reviewed it. Once the article is printed, it can not be updated. Corrections have to happen in a follow-on article, which may not happen and may not be evident when you are looking at the original. Exact customs vary by field, but this is the general pattern.
What happens when a popular Wikipedia article is created? The birth of a Wikipedia article on a high-profile topic is a beautiful thing to witness. For example, Brian Keegan notes that in the 100 hours after the Sendai earthquake and tsunami in Japan, 1,727 people made 6,931 edits to 49 relevant articles. The main Sendai quake page at the time of this writing has 289 references. Everything about it has been checked and rechecked. Today 349 people have the article on their “watchlist”–the list of pages they monitor for changes. (Not everyone actually checks their watchlist of course.) Vandalism on Wikipedia is typically removed quickly–Fernanda Viegas and Martin Wattenberg found that it is often corrected in seconds.
Next time someone is nominated to the US Supreme Court or becomes the next Pope, watch their page as it evolves. On any Wikipedia page, you can click the “View History” tab and see all the edits to the article over time. Over the period of a few days, the newly famous person’s page evolves from a few sentences to a complete concordance on their life and work–with every fact supported by references, and anything unsupported removed quickly. It’s astonishing to witness.
So what would you rather have–something checked by three experts over six months to a year, or something checked by 1,727 people in the first 100 hours? And remember that many of those 1,727 people are checking references and not allowing anything that isn’t documented. Also remember that the refereed journal article is fixed at a moment in time, and beyond that any errors or new developments aren’t included. A Wikipedia article is updated continuously. Of course the purpose of a journal article and encyclopedia article are entirely different–one presents new knowledge and the other summarizes consensus and explicitly forbids original research. They’re not comparable. But if you believe that reliability of knowledge is in relation to how many people check it and how carefully, then a popular Wikipedia article does pretty well. Amazingly well, in fact.
Particularly surprising to me is the fact that topics on controversial issues can be quite good. For example, I would have guessed that the article on whether vaccines cause autism would be a cesspool of controversy and misinformation. But it’s not. It reviews the history of the controversy in comprehensive detail (supported by references) and unequivocally says that the original paper suggesting a connection has been proven a fraud and there is a scientific consensus that there is no such link. Hooray!
But those are all examples of high-profile articles. What about low profile ones? Click the ‘random article’ button on the left hand column of the Wikimedia software, and see what you get. Often you’ll get something that’s barely been started–a “stub” in Wikipedia parlance. It’s possible to put something unsupported in an obscure article, and it may not be checked. Famously, a prankster wrote that journalist John Seigenthaler was a suspect in the assassination of John F Kennedy. The error remained there for over six months until a friend of Seigenthaler’s noticed it. I should note that this happened in 2005 and the culture of Wikipedia has changed since then–things are now checked more carefully. But is it possible for a prank or honest error to linger? If it’s in an article that is not high profile, absolutely.
Another problem is circular references. It happens like this: a journalist uses Wikipedia as a source for some information, and publishes an article without having any other source. Later, a kind Wikipedia editor notices that the article is unsupported and searches for good references–and finds the journalist’s article and cites it! (Oops…)
It’s not surprising that people are confused about whether to believe Wikipedia–the truth is complicated. I believe today we have a crisis in epistemology–no one knows what to believe any more. But it’s also a teachable moment. A moment to teach students about peer review and the importance of references and how to think critically about the reliability of everything they read.
At CHI 2011 in Vancouver, Nick Yee and colleagues presented a fun note about gender swapping on World of Warcraft: “Do Men Heal More When in Drag?
Conflicting Identity Cues Between User and Avatar.” There are many gender-based stereotypes about people’s behavior on MMOs. For example, people assume that women are more likely to play healing characters. However, Yee found that women are in fact not more likely to heal than men. However, men playing female primary characters are more likely to heal a lot. When gender swapping, players live up to their stereotype of gendered behavior, even if that stereotype is not true.
This fascinates me, particularly because Josh Berman and I found the same thing in our study of an online identity game we created in 1999, The Turing Game. In the Turing Game, a panel of players pretends to be a particular identity–for example women. (Game types were user created, so people played lots of fun games like who is from Canada, who is under 30, who is a parent, and more.) The audience asks questions, and votes on who they think is telling the truth. After the game, contestants reveal their real identities and discuss how everyone knew the truth or was fooled. In many of these post-game conversations, audience members would say things like, “I knew you were really a woman, because you use long sentences with lots of dependent clauses. Women talk a lot. Men say things like ‘I’ll be back.’” The only problem with this is that Susan Herring can conclusively show you that men use more words per conversational turn online. The stereotype is wrong. And just as Yee found 12 years later, gender swapping reinforces stereotypes.
We are currently redesigning and reimplementing The Turing Game as a web game with Facebook Connect. Our challenge in the redesign is to figure out how to help these playful explorations yield deeper insights and less bandying about stereotypes (whether true or false).
- Why LinkedIn is Creepy: Asymmetry of Visibility
- A Great Experience That Must Stop: Words With Friends and the Mindful Use of Technology
- The Limits of Crowdsourcing: A Coming of Age
- Artifacts Have Politics. Now What?
- Everything is Cheaper on Amazon--Is This Good News?
- The Online Cheating Arms Race
- How much time are you spending on your smartphone?
- Saturday Morning Cartoons and Other Choices About Technology Use--Thoughts on Sherry Turkle's "Alone Together"
- When are video games mimetic?
- Research Topics: The PhD Student's Passion or What We Have Money For?
- May 2013
- April 2013
- March 2013
- February 2013
- January 2013
- December 2012
- November 2012
- October 2012
- September 2012
- August 2012
- July 2012
- May 2012
- April 2012
- March 2012
- February 2012
- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- 41,642 hits
- Using science to reform toxic player behavior in League of Legends ars.to/15EenzB via @joseZagal 3 days ago
- @fkschmidt thanks Ken :) 4 days ago
- @MikeTufekci glad you liked it! yes--lots of design to do :) 5 days ago
- @MikeTufekci really! Older versions don't do that. That'll help…. 5 days ago
- On Google Glass and Gargoyles: a Call to Action wp.me/pMtBi-iN 5 days ago