As many readers may already be aware, my father is an English professor, and commonly when we get together, we talk about the problems he sees in his classes. One problem which seems to be coming up more and more these days is that his students have difficulty evaluating the validity, usefulness, and source of information. This may be because they are not getting a chance to learn and practice this skill in school, but I think it may have more to do with the fact that the way in which information is presented is shifting, and as a result, it is getting harder to evaluate information.
So, I thought I’d write a bit of a primer, because it’s important to have this skill, whether you’re a student of English, an activist, or just some person who reads the news. For a lot of you folks, this information is really basic, and may be information that you are already well aware of. But you may know people who can benefit from a basic primer on evaluating sources. You may also have some points to add to this rather topical discussion.
Information comes in a lot of flavors. Learning to identify the flavor of information is important, and perhaps one of the first steps to take when evaluating material.
Some of the best information, in my mind, comes from primary sources. Let’s say you want to learn about concentration camps. What better a source than recollections from concentration camp survivors, physical evidence from the camps such as camp records and belongings of occupants, testimony from people who worked in the camps, and testimony from people who came into contact with inmates/employees at the camps? The great thing about a primary source is that there is no remove: the information you see is real and immediate.
In a secondary source, someone is taking information from primary sources, synthesizing it, and writing about it. A great example, to continue our concentration camp example, might be a book about the Holocaust from a historian who took advantage of archives and other primary source material. A good secondary source should include a complete list of the primary sources used, allowing people to seek out the information to confirm it. Secondary sources tend to be less reliable because the material is filtered through a lens, even when the author claims to be objective. As long as you are aware of the lens, though, such sources can be very valuable, especially when primary source information is very hard to access.
Tertiary sources are built on secondary sources. They become even murkier, because rather than relying on primary sources to develop ideas, they are depending on secondary sources, assuming that those sources are accurate, and making extrapolations from those sources. The tertiary source point is the moment when information starts to feel more like a game of telephone than a factual accounting. Again, reputable texts list their sources, allowing readers to judge the veracity of the information for themselves.
One should also think about the qualifications and the credibility of the source.
A historian who has graduated from Oxford with a degree in Holocaust Studies is a better authority on Holocaust matters than a blogger with no clear qualifications. This means that when contradictory information between the two sources is being weighed, the side of the Oxford grad should probably be more seriously considered. It is important to distinguish, here, between primary and secondary sources.
A blogger who acts as a primary source, say, documenting something going on in ou community, is actually a great source. This information may not be available through other avenues, and that makes the blog entries valuable, especially if a reader can discern the blogger’s bias. Bloggers who are secondary or tertiary sources? Not so much, especially if they do not provide citations for their evidence. So, say, a blogger who did not go to a town hall meeting, but interviewed people who did and read blogs written by people who did and provides all of this raw data at the end of a post about town hall meetings? Is a good source. A blogger who writes some vague, rambling diatribe about town hall meetings with no sources? Not a good source.
Hand in hand with credibility goes the force behind the information. My father’s always told me to follow the money, and it’s sound advice for pretty much anything you can imagine.
Let’s say that you are trying to get information about how frequently grain exported to the developing world as part of relief efforts is contaminated. Who is a better source of information: the World Health Organization, the World Food Program, or an agricultural company which sells/donates grain to programs which bring grain to developing nations which need assistance? In this case, the ag company is clearly a bad source, because it has a vested interest in downplaying contamination reports, since this makes it look bad. The World Food Program is a better source, but not a great one, because it also does not want to admit that it may have distributed contaminated grain. The World Health Organization, on the other hand, has no interests one way or the other, and is a reliable source, because it simply provides the raw data in the form of how many people became ill as a result of contaminated grain, or how many lots of contaminated grain it identified during food safety inspections. Ideally, you should collect data from all three, and compare it. Actually finding the data on your own is much better than reading someone else’s collection of the data.
I’ve give you a hint, and something which many people seem to have trouble grasping: the more money behind a source, the less reliable it is. The thing about money is that people who have money want to protect it and make more money. This means that when a lot of money is put behind something, you can darn well bet there’s a reason for it. This is especially problematic in cases of astroturfing, in which major companies hide behind “grassroots activists” who have really been groomed by these companies.
Always. Follow. The. Money. If a report says it’s from “People for Greenfield Preservation,” find out who administers that group. Find out where they get their funding. Critically, find out if they are actually part of a larger organization. Do you think the tone of the information changes if “People for Greenfield Preservation” is owned by the Sierra Club vs Royal Dutch/Shell? Even if the information seems relatively basic and simple, like “70% of the world’s greenfields are in threat due to development,” it’s probably not. How is the organization defining “greenfield”? How about “development”? And, critically, how is it collecting data?
Which brings us to another issue, which is specifically how to look at statistics and scientific data. I don’t want to get too in-depth here, because this is a complicated topic (enough for a whole separate post), but the takeaway is that people can and do manipulate math and science to their own ends. Just because something is a statistic does not make it true.
So, when you see the results of a study, the things you need to be asking include: Can you find a clear definition for what was being studied? How big was the sample group? Is the sample truly random? How did they control for variables? Who oversaw the design, planning, and implementation of the study? Which organization/s backed the study? Who funded the study? Are errors openly disclosed and discussed? Who has their name on the study? Who stands to benefit from the results as stated? Where was the study conducted?
Take a relatively simple question, like how many people in the United States are people with disabilities. You are going to find a range of answers, depending on survey methods, the definition of “disability,” the organization conducting the study, and so forth. Good science should stand up to rigorous examination, so not be afraid to shred a study in search of the information at the core. If the science was done properly, your shredding will not alter the validity of the end results.
I also want to talk briefly about how to find information.
The short version is that you need to find it yourself.
Unless you are talking to a reference librarian, you should not ask other people to provide you with information. Why? Because they are biased. When someone says “you should boycott this company” or “this activity is bad for you” you should seek out verifying information to back that claim. And you should seek out information to disprove it. You need to do the legwork to find out whether or not someone’s assertion is true. If someone makes a direct claim, like “this study says that X percent of Y are Z,” then it is appropriate to ask “could you please point me to the study,” but that’s it. Do not use other people as reference librarians, because they will provide you with skewed, slanted, and erroneous information. Someone who claims to have a “complete list of sources” on a matter is full of the proverbial.
I do want to make a distinction here: obviously, when someone is trying to make a point or assertion, it is appropriate to say “do you have sources to back that up?” This is, in fact, encouraged (because you are trying to find the source of this person’s information). But asking someone to lead you to information? Not appropriate. People can provide you with the sources they are using, but they cannot provide you with complete sourcing on the matter. That’s your job. You need to evaluate all of the data and come to your own conclusions, and you can’t do that if you are relying on someone else to generate that data for you.
So, why trust a reference librarian? Because reference librarians are trained to look for information, so they are really good at it. They also don’t have a dog in the fight. They don’t care, one way or the other, what the information proves or doesn’t prove, and they may in fact prefer it if people just say “I’m looking for information about this company,” rather than “someone told me to boycott this company, is there information about it?” A reference librarian will happily dig up a wide variety of resources for someone who does not have the skills or the time to find data independently. Reference librarians, in other words, are your friends.
This concludes this extremely rough and topical discussion about information and how to look at it. I am well aware that I left a lot of things out. But this should be a start, and most certainly is not intended as an authoritative document. I can’t teach critical thinking in 2,000 words, after all.