Should you believe Wikipedia?
I got a nice email today from librarian Tedd Guedel of Herzing University asking about the reliability of Wikipedia (he saw the announcement for a talk I gave last October, ““How Wikipedia Really Works, and What This Means for the Nature of “Truth”.”) With Tedd’s permission, I quote:
“As a rule, I steer students away from Wikipedia as a valid academic source. Do you have a power point or any information that you gave during your lecture that you could share with me? … Our old IT director Loved Wikipedia because “it is maintained by the masses” I do not like Wikipedia because “it is maintained by the masses.”
So who’s right, Tedd or his IT director? Luckily in this case I don’t have to take sides–they both are right. The answer is: what page on Wikipedia? How many people have edited it? How many people are ‘watching’ it? I will argue that a popular, high profile Wikipedia page is the most accurate reference that has ever been created in the history of the written word. (Really!) A low-profile page that few people have edited is unreliable. It all depends on how many people have checked the article and its references.
To explain why, it might help to discuss how a refereed journal article is reviewed. Articles in high-quality, peer reviewed journals are generally considered the gold standard for reliability. An author submits an article to a journal, and it is sent out to approximately three experts in the field for review. Each expert reads the paper carefully, and sends detailed comments. The experts are anonymous to the author, and can make critical comments without fear of giving offense. The author usually revises the article, and the experts read it again. Most articles are rejected. Nothing is published until the experts are happy. The reliability of the article comes from the number of people who have reviewed it, the special expertise of those people, and how carefully they reviewed it. Once the article is printed, it can not be updated. Corrections have to happen in a follow-on article, which may not happen and may not be evident when you are looking at the original. Exact customs vary by field, but this is the general pattern.
What happens when a popular Wikipedia article is created? The birth of a Wikipedia article on a high-profile topic is a beautiful thing to witness. For example, Brian Keegan notes that in the 100 hours after the Sendai earthquake and tsunami in Japan, 1,727 people made 6,931 edits to 49 relevant articles. The main Sendai quake page at the time of this writing has 289 references. Everything about it has been checked and rechecked. Today 349 people have the article on their “watchlist”–the list of pages they monitor for changes. (Not everyone actually checks their watchlist of course.) Vandalism on Wikipedia is typically removed quickly–Fernanda Viegas and Martin Wattenberg found that it is often corrected in seconds.
Next time someone is nominated to the US Supreme Court or becomes the next Pope, watch their page as it evolves. On any Wikipedia page, you can click the “View History” tab and see all the edits to the article over time. Over the period of a few days, the newly famous person’s page evolves from a few sentences to a complete concordance on their life and work–with every fact supported by references, and anything unsupported removed quickly. It’s astonishing to witness.
So what would you rather have–something checked by three experts over six months to a year, or something checked by 1,727 people in the first 100 hours? And remember that many of those 1,727 people are checking references and not allowing anything that isn’t documented. Also remember that the refereed journal article is fixed at a moment in time, and beyond that any errors or new developments aren’t included. A Wikipedia article is updated continuously. Of course the purpose of a journal article and encyclopedia article are entirely different–one presents new knowledge and the other summarizes consensus and explicitly forbids original research. They’re not comparable. But if you believe that reliability of knowledge is in relation to how many people check it and how carefully, then a popular Wikipedia article does pretty well. Amazingly well, in fact.
Particularly surprising to me is the fact that topics on controversial issues can be quite good. For example, I would have guessed that the article on whether vaccines cause autism would be a cesspool of controversy and misinformation. But it’s not. It reviews the history of the controversy in comprehensive detail (supported by references) and unequivocally says that the original paper suggesting a connection has been proven a fraud and there is a scientific consensus that there is no such link. Hooray!
But those are all examples of high-profile articles. What about low profile ones? Click the ‘random article’ button on the left hand column of the Wikimedia software, and see what you get. Often you’ll get something that’s barely been started–a “stub” in Wikipedia parlance. It’s possible to put something unsupported in an obscure article, and it may not be checked. Famously, a prankster wrote that journalist John Seigenthaler was a suspect in the assassination of John F Kennedy. The error remained there for over six months until a friend of Seigenthaler’s noticed it. I should note that this happened in 2005 and the culture of Wikipedia has changed since then–things are now checked more carefully. But is it possible for a prank or honest error to linger? If it’s in an article that is not high profile, absolutely.
Another problem is circular references. It happens like this: a journalist uses Wikipedia as a source for some information, and publishes an article without having any other source. Later, a kind Wikipedia editor notices that the article is unsupported and searches for good references–and finds the journalist’s article and cites it! (Oops…)
It’s not surprising that people are confused about whether to believe Wikipedia–the truth is complicated. I believe today we have a crisis in epistemology–no one knows what to believe any more. But it’s also a teachable moment. A moment to teach students about peer review and the importance of references and how to think critically about the reliability of everything they read.
The Next Bison: Social Computing and Culture 
This is really illuminating and very helpful. Thank you!
One of the things I like about Wikipedia is the fact that the articles don’t remain static. We teach a course called Theory of Knowledge at our school, where the purpose is to help kids understand “how we know things.” The fact that Wikipedia changes, and sometimes rapidly enough for a class to dissect on the moment, suggests that knowledge is not an immutable thing, that it a dynamic growing endeavor.
Amy, funny thing, I just blogged about why I’ve changed my tune and now encourage my students to use Wikipedia. Here it is http://bit.ly/k16s56
Your post is much more convincing than mine. Thanks for the excellent information!
I like yours too! Thanks for the comment!
Would it be possible to automatically rate the reliability of an article based on metrics like you describe (number of edits, number of watchers, etc), and if so, would this be a useful feature for Wikipedia to add?
@John Pane: It would likely be possible, but as a Wikipedian, I’d oppose such a measure being implemented. There are too many holes in raw statistics on an article for any particular metric to be useful. For example, lots of editors could also mean that the article has been subject to vociferous and damaging edit wars, and lots of references might be dominated by low-quality sources as opposed to the nicer ones that we prefer. The best metric that we tend to have is the manual WikiProject rating system: Stub, Start, C, B, A, Good Article (GA), and Featured Article (FA), ascending in roughly that order (GA & A might be inverted). I’m writing this on my iPhone, so I won’t provide a link, but google “Wikipedia 1.0 assessment” and you should find a full description.
John and Nihiltres, interesting discussion! I wonder if anyone has ever done a formal comparison of raw stats vs wikiproject ratings vs more sophisticated metrics like Luca de Alfaro’s. Luca’s method relies on the aggregated reputations of the authors of an article. Where reputation is related to how much of the text you’ve contributed has been reverted. I think there’s more good work to do here!
Really like this article. Helps me understand the whole process for publishing on Wikipedia. I have told my Adult School students that Wikipedia is a great place to start research, often because of the outline, and it gives a good overview of the topic. I tell them to then check out the references and use those as places to go to for original research and come to their own conclusions. But, I also tell them that Wikipedia is usually not accepted as a reliable source at most schools. So I say use it, but don’t cite it.
I’m sorry, but you are entirely wrong.
Wikipedia is entirely constrained by the biases of its contributors. Take for example Timor Leste – the article is infested with Australian and United States views of the country. Many of these are questionable, a number are wrong. An encyclopedia fails in this way, because its use is to provide a summary to an audience. A good reference work tells you specific things in detail.
As an educator I forbid my students to use it for scholarly research. I tell them to read whatever article first, but then follow the references.
So what would you rather have–something checked by three experts over six months to a year, or something checked by 1,727 people in the first 100 hours?
On topics that require specialist knowledge, the former without hesitation.
I found this to be very informative, thanks
So what happens when the contributor puts down false information? Piers Morgan claims to have been born in 1965, which would have made him 13 years old when he started free-lance writing for the tabloids. Hardly likely!
Frederick Forsyth claims that everything written about him in Wikipedia is false.