College Quarterly
Winter 2008 - Volume 11 Number 1
Home

Contents
The Critical Evaluation of Bibliographic Web Sources

By Jim Gough

Introduction: The Problem

With the rapid increase in information freely and easily accessible on the web to those who have access to a computer and the internet, there seems to be a corresponding decrease in critical evaluation of the sources of this information. All sources are taken to be sources of information and seem to be uncritically considered to contain reliable, credible, and authoritative information, which is relevant to any topic. However, many sources contain disputed and disputable information slanted to support one ideological belief over another. People use sources in their research bibliographies which are concurrently used to support claims in their research papers that are not in fact plausible. Students, as citizens, need to be critically informed to make good decisions in a democratic society that depends on their reliable, credible and authoritative sources of information used in decision making. While students seem aware of the formal aspects of MLAd or APAbibliographic formatting, they are unaware of their individual responsibility to employ reliable, credible, authoritative sources. Not all sources are acceptable. So, what is needed is a set of testing conditions to determine how to critically separate the acceptable sources from the implausible ones.

For example, while a plausible-- arms length-- government web site (Statistics Canada, www.statscan.ca) provides information that the murder rate in Canada, factoring in population and other variables, has dropped by approximately 20% over the last forty years since the elimination of the death penalty, a student recently submitted a critical paper citing an increase in the murder rate of over 25% during the same period of time. The source used by the student was a site created by a group whose only espoused goal was to bring back the death penalty in Canada and who provided no reference to the actual research used to create the claim to a 25% increase in this crime (www.cap-crime-stats.com) . The student accurately cited his internet source, a source which he used as the main support for his argument that Canada should return to the use of capital punishment. However, his use of this source was unacceptable because the information was not reliable, biased and not credible. Why did he make this mistake?

First, this mistake may be due to a confusion built into the access to this information. Much of the information on the internet is intended to sell the information seeker something, a product or a service, not simply to inform him or her of some set of so-called facts or provide access to neutral information. So, the line between a pure information source and a sales pitch with potentially biased or self-serving information about a product or service is often blurred. Often I ask: Have I accessed an on-line department store or a library? One student looking for information on whistle blowing found a site in which a law firm was giving information in order to advertise the success it had at winning whistle blower law suits and the site provided little understanding of whistle blowing (www.whistleblowers.com). This blurring of the borders can create a situation that is detrimental to the critical evaluation of information, a confusion about the source and the kind of information provided.

Second, because internet sites are easy to access and so widely used, these traits (ease of access and wide use) take precedence over the depth, coherence or accuracy of the information provided. Type “dictionary” into the Google search box, for example, and you will get one but it may not be the most reliable or authoritative dictionary. Internet sources promote the number of “hits” on their sites. This is a version of the informal fallacy of popularity—the mistake of arguing that the more popular a site or source of information, the more credible the site. Many, but not all, of these internet sources do not provide students with accessible alternatives or sites with contrary information to the source’s claims. Site links are often to other sites with favoured views. Many sites claiming knowledge of intelligent design favour this ideologically biased strategy (sources too numerous to mention can be found by simply web siting “Intelligent Design”). This is very much unlike the physical conditions in a library where on one and the same shelf, for example, there may be texts containing arguments supporting capital punishment, texts rejecting the ethical viability of a society supporting capital punishment, and even texts providing both the pro and the con arguments. This diversity of views challenging each other on one location is rare on the internet. Some internet sites are determinately promotional for a negative effect on a student’s ability to function open-mindedly towards value issues. Readers need to be wary of agenda statements which promulgate claims that unnamed others are attempting to distort, confuse or misuse scientific evidence that should favourably support their ideology or agenda. For example, a global warming denier site (Global Warming: A Closer Look at the Numbers, Monte Hieb, January 10, 2003) contained charts and graphs claming that global warming was caused by the unrecognized, but not humanly caused, increase in water vapour—not the increase in carbon dioxide, methane or other gases. While citing sources of his own, this site’s creator did not identify his own credentials or institutional source, while clearly confusing in his site the difference between a cause and an effect: an increase in water vapour is in fact an effect of global warming (not a natural cause of it) caused by increases in C02 emissions into the atmosphere—not, as the site claimed, a cause of global warming.

Third, there is a fact and fiction confusion in some post-modernist approaches that infects the use of the internet by students. Basically, the view seems to be that all facts are a product of someone’s interpretation and that the use of imagination in this interpretation is not only useful but essential, making comparisons between an interpretation and the factual situation it is based on, functionally impossible. For example, a well-know and popular science fiction writer, Michael Crichton, created a document on his web site promoting the idea that environmental issues, like global warming, were fabrications of a particular vested interest, self-serving, political movement similar to the attempts to establish women’s equality in American federal law. . The problem with this site was that (i) the document created on it looked like a fallacy filled argument rather than the expression of one individual’s personal opinion, point of view, or preferences and (ii) the document used sources, including graphs and citations of studies, which were all creations of the author’s imagination and not based in any scientific evidence . As a fiction writer, he is clearly free to promote his particular point of view but that this view is portrayed as a scientific argument complete with fabricated or invented evidence is problematic. There are a number of such sites which also are questionable because they are not strictly giving scientific information, but more often promoting a political/ideological agenda. The general areas of such sites are controversial issues such as: creationism, intelligent design, capital punishment, and religions connections to politics.

Possible Solution to the Problem: Critically Testing Sources

Anyone who uses electronic sources in a research project or paper, web sites for example, in a critical bibliography should be required to take responsibility for sources which are: reliable, credible, authoritative and relevant. Too often there is a disconnect between the responsibility someone takes for the claims made in a paper, essay or report and the lack of responsibility for the source or support for these claims from material cited in the bibliography. In order to determine that a source is relevant, it is first necessary to determine whether or not it is authoritative. If it is not authoritative then it cannot be relevant. However, relevance is a separate test. Even if a source is authoritative (satisfying the reliable and credible probability tests), it may still be irrelevant since its expertise or authority may make no difference to understanding or accepting the issue at hand or under consideration. For example, the authority of a fiction writer is not the same as that of a professional scientist. There are two different areas of expertise involved.

To test for reliability we can start with sources that generally or normatively identify the expertise of the authors of the web site, whether this is a person or an organization. In the latter case this may be a post secondary educational institution, an arms length government organization, or a not for profit public interest group whose research history and scientific procedures are well-known and respected. Unreliable sources can often be identified by the following: failure to identify the sources of expertise (individual or institutional), use of disconnected data or data interpreted in a fundamentally erratic way, diagrams, graphs, charts which are inadequately explained or developed (so that the site author has to interpret the data for the reader), evidence of a political or ideological (often economic) agenda, sometimes in the form of a claimed “conspiracy” as the focus for the site.

To test for credibility requires a critical look for the credentials of the author, something many people do not know how to do or understand well. Some sites are authored by self-proclaimed authorities or well-known personalities. The popularity of such individuals is not an indication of their credibility or expert knowledge. Others claim credibility which doesn’t match their formal expertise. For example, in one web source, a clinical psychologist’s knowledge did not translate into an acceptable knowledge base for the site’s claims about climatology (www.globalwarmingsource.com). . This transfer of expertise is a well-known move by marketers and is often an illegitimate attempt to obscure the difference between popularity and credibility. Credibility is established within some area in which expert knowledge is possible and tested by knowledgeable practitioners. Someone without any formal education may have knowledge in some area but how could anyone know with any degree of certainty? The professionalization of specialized knowledge is intended to insure credibility in the process of deciding who is an expert. So, testing for credibility involves asking critical questions about the credentials of the individual who gathered the information, how the information was gathered, how the information was evaluated by the practitioner herself, who we assume knows how to use reliable data gathering techniques, other experts in the area (accredited peers in the knowledge discipline), editorial reviewers or a review board.

While experience-based testimonials from users of some device or process are interesting, they are often not founded on reliable testing procedures and can be the product of rationalization or wishful thinking. Credible expertise has to be open to being tested, even if the reader is not explicitly aware of whether, in fact, such testing occurred. There is a well-established practice of using web sites to disseminate information that could not –in principle or in practice –be vetted through other more reliably tested sources, like reputable newspapers, magazines, journals, and encyclopedias. Rumours can be reported in newspapers, whose source is an unreliable web site. Time permitting, readers should be encouraged to follow analogical and skeptical procedures using information from one web site and comparing this information to other web and non-web sources.

To determine the authority in a web site involves the satisfaction of the reliability and credibility tests which sometimes involve the same discrimination. So, while the geological research institute is a reliable source of information, the geographer who authors the site or some part of it is not necessarily a credible source because his area of expertise does not authoritatively translate from geography to geology. Whether a web source is reliable may indicate that a source is authoritative while the decision whether it is credible does not. Alternatively, the credibility of the author of the site may be all that is available to decide whether the site is authoritative or not.

Finally, it is necessary to decide whether the authoritative information provided in the source is also relevant to the general topic in the bibliography. So, for example, even if information on the life of the philosopher John Stuart Mill is reliable and credible, it may still make no positive difference to our understanding the implications to the utilitarianism, which he authored. Relevance is the final challenge that must be critically satisfied for the web-based bibliographic source to serve its function as an acceptable and effective piece of information to be used as a potential reference in an argumentative essay.

Appendix A

The following assignment was distributed in several senior level courses to help students understand how to critically evaluate electronically-based research sources.

The Assignment
  1. Identify and critically evaluate four (4) references, for one (1) of the topics listed in 4 below, from the following set of possible sources (do not duplicate more than one source): Textbooks (excluding the course text and similar introductory course texts), Encyclopedia/Reference texts, Websites/Electronic sources, Critical Reviews, Journal Articles, Journal Discussion Papers, Published Proceedings of Refereed and Edited Conferences, seminars or workshops in print or CD-ROMs.

  2. The written critical response for each of these references should be approximately 2-4 double spaced, typed pages in length, or legibly written. Note: This is a guide to the written length of the assignment but it is not the basis for determining the grade on the assignment. The grade will be based on content not word count. Every effort should be made to include relevant information and to eliminate historical background information, padding and repetition, as well as unnecessary use of quotations.

  3. The critical response should include (a) a clear and precise identification or description of the reference source, including title, author/editor(s) , place and date of publication, publisher or group that supports the site or authors the WebPages, and (b) a brief description of the content, (c) an evaluation of the value, utility or use of the source in terms of its reliability, credibility, as a piece of authority, in providing support in an argument as well as (d) a decision or judgement about the relevance of the source for any critical research, in terms of whether it could or could not enhance our understanding of the topic, makes it clearer or more precise, or increases our appreciation of the issues or problems raised in the topic area. Note: It is not necessary that you be absolutely certain about the relevance of the material found to the topic. It is only, minimally, required that you provide some reason for believing that it is relevant to the topic area. If it is not relevant, then this is still a useful piece of information which, for purposes of this assignment, should be included in at least one of your examples.

  4. Focus your critical bibliographic research [described in 1-3, above] on (1) one of the following general topic areas: (a) Professionalism and Individual Ethical Responsibility, (b) Codes of Ethics and Effective Ethical Decision making…
Problems Successfully Completing this Assignment

There is a general problem produced when people veer or shift away from the assigned tests, in a red herring distraction, to determine the evaluation of a source to other factors not identified in the assignment, factors such as: whether the article was easy for the student to read and comprehend, whether the article contains bias, opinions or valid claims, all of which are not defined by any of the assigned testing procedure. These features of the source have no defined test and they are not necessarily relevant to determining whether a source is acceptable. For example, some students focused on the currency of information, which is important for some ideas but not for every piece of information, especially historical interpretations. As well, some students thought that the location of the source made a significant difference when the topic was “global warming”, indicating that an American source of information was less reliable than a Canadian source. However, the topic was “global warming”, not “global warming from a Canadian perspective”. The authority and relevance of a source does not depend on (i) anyone’s subjective degree of difficulty or ease in reading it, (ii) the latest date when it was updated, since updating a piece or mis-information with more mis-information is not relevant.

The library resource should not be restricted to a computer terminal, despite its advantages. Every researcher should be encouraged to compare hard copy texts, articles and reviews, as well as web sources to gain a good sense of what is or is not a credible and reliable piece of authoritative information relevant to a specified topic.


Jim Gough teaches Philosophy at Red Deer College in Red Deer, Alberta. He can be reached at jim.gough@rdc.ab.ca

Contents

• The views expressed by the authors are those of the authors and do not necessarily reflect those of The College Quarterly or of Seneca College.
Copyright ©
2008 - The College Quarterly, Seneca College of Applied Arts and Technology