Lesson 2 - Ranking System for Evaluating Articles (Activity 3)
Articulate a clear set of criteria that you would use to rank the quality of educational .resources.At the end, identify the top 2 and the bottom 2 types of resources based on the list below.
Then find someone who has a different perspective than you, and try to convince them that your approach is better.
- Book (e.g., Tapscott – Grown Up Digital)
- Chapter (e.g., Classroom Response Systems)
- Journal Article (e.g., Laptop Effectiveness)
- Conference Paper (e.g. MOOCs)
- Report (e.g. MOOC Report)
- Web-based article (e.g., To Mooc or Not to Mooc)
- Blog Entry (e.g., When Helping Hurts)
- Magazine/Newspaper Article (e.g. New York Times – MOOC)
I have outlined the criteria I use when evaluating sources. I have ordered the criteria based on what I believe to be their importance.
ReplyDelete1) Peer reviewed - peer reviewed ensures that the quality of the source is high, having had to pass through a number of colleagues to get published.
2) Location - where is the source published? Different journals or publishers may have different criteria that an author has to meet.
3) Purpose - what is the purpose of the source? Was the source written to educate, to convince etc. Additionally, some sources were requested/sponsored by organizations or governing bodies. The source’s purpose must take into account the purpose of the organization or governing body.
4) Author and their background - Authors and the credentials play an important role in evaluating sources. Obviously, authors who write in their subject field will have more credibility.
5) Year - finally the date the source is published must always be taken into account. It is important for sources to be up to date with current research.
Using my ranking system I identified the Journal Article “Laptop Effectiveness” and the Chapter in “Handbook of Research on New Media Literacy at the K-12 Level” as the top two sources. The bottom two sources are the Web-based article “To MOOC or Not to MOOC” and the Blog entry “When Helping Hurts”.
Hi Gillian - I have a few questions. Why is location important? Do you mean reputation or the quality of criteria used to evaluate resources? Also, regrading author and background - How would you evaluate that and what about "new" authors or professors who do not have a track record? Finally what if criteria conflict? So you have an expert author publishing the MOOC article?
DeleteHi Gillian,
DeleteI am not really sure, what you mean by the location of the publication. By purpose, do you mean that if the paper has enough evidence to prove a claim or is it just an opinion based paper?
In order to rank each resource, an evaluation of the information within each resource must be considered.
ReplyDeleteI began by researching how others recommended to rank resources for academic use. After reviewing a number of sources I found an article I found most closely aligned to my thoughts and thus I determined to rank the resources by considering five criteria including authority, audience, accuracy, objectivity and currency (Finding and Evaluating Research Materials - Research Tips - University at Buffalo Libraries, n.d.).
I created an excel spreadsheet to keep track of each resource and its strength or weakness based upon each of the five criteria. I created a score of 2,1 or 0 to score each resource by each criteria. I then looked at each hotlinked resource example and scored that resource based on this ranking score for each critera.
Once I had scored each resource by each criteria i then totalled the scores for each resource to determine the final ranking.
In completing this I found the top two resources to be the : Journal and the Chapter
And the bottom two resources to be the : Blog and Web based Article
Reference - Finding and Evaluating Research Materials - Research Tips - University at Buffalo Libraries. (n.d.). Retrieved January 16, 2017, from http://library.buffalo.edu/help/research-tips/evaluate/
Hi Melanie
DeleteI was just curious why you chose a chapter to be a better resource than a book?
HI Adam; For this particular example the book which this assignment linked to was the reason it fell further down my ranking than a chapter. The book was based on a study funded by the company owned by the author of the book. This decreased the "objectivity" score I assigned it enough that it fell below the example of the chapter. I used the examples linked to score so other books may have scored higher up the ranking.
DeleteAhh that makes perfect sense. I am not sure if I was interpreting the assignment wrong, but I thought we were suppose to look at the source in general, so a book chapter in general not that specific book chapter. Oops! Your logic makes perfect sense!
DeleteHi Melanie, Can you expand on what you mean by authority, audience, accuracy, objectivity and currency? Also, do you see any risk in relying on authority?
DeleteMelanie - Make sure you are as accurate as possible with your facts. The study was funded by numerous large companies, not just the author's company. And the author is surprising transparent about which companies and the parameters of the study. Why, in this case, does funding make a difference? What benefit would a company derive from this study? Does the fact that McGraw Hill Education was the publisher make any difference?
DeleteHI - Yes that is a summation and not the sole issue with the ranking. As I have not read the book I was relying on the information outlined on the amazon preview pages which were available. Although multiple companies were highlighted as "beneficiaries" of this research, I did not notice specifics highlighting the funding of the research. I was using the information provided about the research program team which appeared to be compiled of individuals who worked or were subcontracted specifically for this research by the authors company. I may have misunderstood this in reading just the provided pages at a high level and perhaps additional detail in the full print version emerges. In terms of whether the publisher McGraw Hill Education makes any difference I would say for my ranking I did not consider the publisher. Should I explore my ranking system again I might consider this - however that would require me to have more knowledge about publishers than I currently have! :)
DeletePublisher is a consideration - McGraw Hill is a major publisher as is Pearson, so we might assume they are not publishing absolute nonsense. A kindle book on the other hand, or Tapscott Publishing (in this case) might be an issue. No need to change anythind - just pushing your thinking a little.
DeleteHI: This is the way I ranked the resources and the definition of what each factor considered. In terms of the funding it was only one of the variables which affected the books ranking in relation to how I had ranked these resources - You can see more here - https://docs.google.com/spreadsheets/d/1Vg7x1WKadn3H_cahfSiTxxjlOHxEl6i2vGe9f_aVYOY/edit?usp=sharing
ReplyDeleteThanks for sharing your spreadsheet, Melanie. I like your approach! I don't have significant disagreement with your results, but I wonder about a few points in your method:
Delete1) How did you distinguish between accuracy and objectivity? Your 0,1,2 descriptors seem to indicate considerable overlap.
2) Why did you weight audience, accuracy, and objectivity twice as important as authority and currency? That is the effect of having 1 and 2 point rating scales.
When confronted with the challenge of ranking the types of resources for their quality, my first thought was to develop a flowchart with decision branches and incrementing values to determine a rank. I began down this path and made some progress but struggled to establish the intricacies of the decision flow. I think such an approach might be possible. However, it would not be as simple as a coin sorter where sorting just by size would provide a reasonably accurate analysis.
ReplyDeleteIn the end, I created a spreadsheet (https://goo.gl/xb6zHY) with rows for each resource type and columns for each characteristic. This approach reduced the need to be concerned about the exact process. This method also allows for a considerable extension of the system as rows and columns could be created for additional types of resources and characteristics. In my spreadsheet, each feature of each resource was assigned a value of 1 or 0 and all of the values summed to create a ranking. I phrased each characteristic so that a value of 1 was desirable, with the result that the highest total is the highest ranked resource. Using this approach also means that my system can be refined by using a greater range of values (for example, a 1-10 scale) or by adding weighting to the ranking calculation.
The characteristics that I assessed are 1) whether empirical research was conducted by the author; 2) whether the research was peer reviewed; 3) whether the research is recent; 4) whether the resource was professionally edited; 5) whether the resource was intended for a scholarly audience; and 6) whether the resource was professionally published. Based on my assessment, journal articles and chapters are highest ranked and blog posts, and web-based articles are lowest.
Tim,
DeleteI like the setup of your ranking system as well as the spreadsheet you have created. I am interested in the "currency" variable - What does this refer to? I'm also curious as to if you agree with your scoring system after it gave its final result, as "Newspaper/Magazine articles" scored somewhere in the middle for you ranking system. I tend to find a lot of bias in information presented when using newspaper/magazine articles as a resource... They also seldom give their resources so I can dig in to further research.
- Andrew
Fair points, Andrew.
DeleteCurrency refers to the recency of the publication. I could make my spreadsheet more helpful by including more description of what the characteristics refer to and how they were evaluated. I used 5 years as the measure of currency. Obviously there will be exceptions. For example, books and articles that establish the theoretical framework have a longer shelf life, whereas blog posts and newspaper articles about current use have shorter.
Yes, I had wondered about adding additional characteristics to the system to give a lower ranking to "Newspaper/Magazine articles". It might alternatively be a flaw in my analysis. I'm not sure. A characteristic that I couldn't figure out how to include in my system is the intended use of the resource. The secondary use of a newspaper/magazine article might be acceptable, where the primary use of a book might not. I confess that I vacillated between thinking about the resource type and the particular examples provided by Robin. That may have resulted in my system being less precise than I had hoped.
Hi Tim,
DeleteI like your flowchart idea! But, like you, I would find it limiting and inflexible. There are simply too many ways (some of them very subtle) that we evaluate the credibility and value of a given resource. Something as subtle as a spelling mistake can change the level of confidence I have in a literary work. Tone, word choice, and even punctuation are all subtle thing I consider when I read. They might not all be columns in my spreadsheet, but they are things I notice...
In addition to the columns in your spreadsheet (well done, btw) I would add a column for publisher or author bias. Or perhaps more generally worded for "purpose" of the publication. Is the paper trying to make you agree with it? Does the author or publisher benefit (usually financially but not always) if the reader agrees? Does it show BOTH sides of the topic/question?
Thanks for the post and for sharing your spreadsheet!
James
@James - I think actually want to move away from the "subjective" way of analyzing an article and move toward something a little more objective. I understand exposing our personal biases, but perhaps we can move beyond that? Spelling is not often an issue NUT journal author's are notoriously awkward and poor writers, so I think you might want to move past that. Tim has outlined a pretty extensive list of criteria that work pretty well. The purpose of the author is important BUT Time may have covered that in the "Is the purpose academic/scholarly"?
DeleteHI Tim; Thanks for sharing your system. I like your considerations. I wondered if you had a different rank for professionally edited and professionally published in terms of "who" the professional is? Thanks
DeleteSorry - this posted before I finished writing.. I ask this as publishers exist for different industries/areas etc and I wonder if your thoughts may change the ranking of a resource that was well referenced but perhaps published or edited by a group/company/individual who is not necessarily known?
DeleteHi everyone! I thought I had posted my ranking of reliable/less reliable but I don't see it here... I will link to a google doc instead as to not take up too much room on this thread:
ReplyDeletehttps://docs.google.com/document/d/17FKvGBeiHmFH18xrMXYlUEIMLwHJmX31WLm9fWjymtI/edit?usp=sharing
Thanks!
Andrew
Hi Andrew,
DeleteWhere, in your ranking system, would you rank "currency" (meaning how accurate the information is right now). You ranked books as the best quality of resource, yet books can often month or even years to get from authorship to the shelves of Chapters. Given that we are in a digital technology program, and it is understood that digital technologies change at a rapid rate, do you feel that a book is current on the day you pick it up off the shelf?
The process I followed for this assignment was to identify my criteria for critically evaluating the quality of a resource, then I ranked the 8 media types against those criteria. I am interested to know what criteria you chose and where currency ranked on your list.
Thanks!
James
Andrew's System
DeleteBooks
- Extensive information and research, usually detailed and includes references for information cited
- Usually a reliable resource due to the time and effort put in to the book
Chapter
- References clearly defined and cited
- Generally reliable (similarly to journal article’s and books) due to the time and organizational structure spent on putting it together
- There are usually several editors of the handbook being put together, which provides some added security that the information you obtaining is accurate
Journal Article
- Specific, reliable information about topics that are up-to-date and are frequently published (compared to a book which might take years to compile and release)
- References clearly defined and cited
- Usually has a set framework/structure
- This resource is reliable, similarly to books, due to the extensive time put in to the process
Conference Paper
- Can have useful information but may be more based on ‘expert opinion’ as opposed to information backed up through references
- Reference lists may not be as extensive as those found in chapters/journal articles
- Process may not be as rigorous – research might be ongoing and so its findings may not be definitive
Report
- May be focused or driven by a problem or field of interest in the workplace
- Focus may be to recommend actions or solve problems as opposed to researching with an open mind/no agenda
- Can be reliable, but is important to look at the references and information presented
Web-based Article
- May be unreliable – could have no author, website publishing it may have an agenda for why they’re publishing
- Often does not include references to research
Blog Entry
- Most blogs are likely unreliable as people present information as opinion (rarely do I see information cited)
- Newer blogs, created by university students, is starting to change this trend
- It’s important to analyze the blog to see if the source seems credible (is it a Masters student? Professor/Researcher at a University? High school student?)
Magazine/Newspaper Article
- Similarly to web-based articles, the resource may be unreliable
- Information rarely cited
- Bias may be present based on the publisher
Based on the above, my top two resources are: Journal articles and books
Based on the above, my bottom two resources are: web-based articles and blog entries
Not related to my submission, but after reading some of our posts in here I went looking for information about evaluating media. I found an article link that might be of interest.
ReplyDeletehttps://goo.gl/wpPHlb
See you all soon!
Thanks,
James
Hi James
DeleteThanks for posting this. A lot of what the article said I had considered, but it raised multiple questions that had not even entered my mind when considering these sources.
Hi James,
DeleteWould you share your reasoning for placing the conference paper ahead of the chapter? I thought the chapter was a better quality resource because it contained research study results whereas the conference paper seemed more general in topic.
Hello Everyone,
ReplyDeleteI've posted my ranking couple of times and I don't see it here.
I am posting it as a google doc thread:
https://docs.google.com/document/d/1oDyH6NsOTfmLnZurRCNhWWE6WNIMrURuZPgZGs0rgUE/edit?usp=sharing
Cheers,
Nirusha
Nirusha's System
Delete1. Journal articles from highly ranked journals: Most current reliable resource. All claims are supported with evidence.
2. Book: Not very current. But, reliable information presented with supporting data and detail background information on the topic.
3.Chapter: Often not up to date. Contains reliable information. But, may not present enough background information and details on the topic.
4. Report: e.g. Ministry reports present reliable information based on scientific evidence.
5. Conference papers: Most often present new ideas and therefore may not contain enough evidence to support the claim.
6. Magazine/newspaper: Often not reliable. Do not present accurate information with supporting evidence.
7. Web-based articles: Often, information provided is not supported with reliable resources.
8. Blog entry: The least reliable resources. The entries are author’s personal reflection on a subject matter.
The journal articles and books are my selection of best resources. The worst quality resources are: Blog entry and web-based articles.
Nirusha,
DeleteHow do you tell if a journal is highly ranked? Why are books and chapters inherently not current? Are there other kinds of reports that are not as reliable/valid?
HI Nirusha; I agree overall with your ranking of the articles however with regards to book or chapter would the date be a consideration as I can agree that some out of date but perhaps new release (or perhaps information on how long it took for the publishing from time of writing) might change your thought on date? Also - I used the examples linked but I also would be interested in your thought if a web-based article was supported by references if that might change your mind. I am curious about this myself with the decline in print media and increase in web based articles and online media.
DeleteIn case my post disappears again, here is a link to my ranking in a Google Doc: https://docs.google.com/document/d/1bRwNRZfHqIy3_gC7MwHyfhOQW7wcwXUm2iRm795nggs/edit?usp=sharing
ReplyDeleteAdam's Criteria
ReplyDeleteI had a lot of trouble thinking about this, because I always seemed to find a counter argument to criteria within my ranking system. Should we consider how recently the information was published as criteria? That makes sense, but if we were researching a principle in mathematics that information could be 500 years old and still completely valid. Should the reputation of the publication come into play? In my opinion yes, but this assignment asks us only to consider a newspaper article in general, not the New York Times versus The Toronto Sun. I think I just wanted to get my thoughts out in the introduction here to vent about some of the challenges I found in the thinking process and see if anyone else struggled similarly. To overcome this, I have put on the hat of somebody researching educational information specifically, and have come up with the following criteria.
Criterion One
The number one criterion I consider is the validity and integrity of the information. Is the source of the information under a legal, moral, or ethical obligation to ensure the validity and integrity of what they publish? Is there a peer review process? If so, I think this means that the information presented warrants a higher ranking than sources who cannot satisfy that criteria.
Criterion Two
The reputation and credentials of the author and publishing company were next for me. To say that all books/websites/blogs/authors are equal in the quality of the information they provide would be wrong. There is a process by which a book is scrutinized before some publishing companies will publish it, whereas other companies are willing to publish anything they believe will make the company money. The same could be true for blogs/websites/chapters/reports etc.
Criterion Three
The recentness of the publication in educational research is an important consideration. That is not to say that foundational research that is decades old is no longer useful, merely that the publication date should be heavily considered when determining the usefulness of information. I believe our literature review is to focus on recently published information, which further solidifies my belief that publication date is an important criterion.
Criterion Four
Finally, the objectivity of the information is essential to me. I know this may run together with previous criterion, but this is perhaps more of a subjective eye test than any of the more objective measures listed above. Does the author or publication have an agenda? Political slant? A bias? Information taken from sources that are not typically objective is not useless, but must be more heavily scrutinized.
Despite these criteria, I am still struggling to rank beyond first place. The clear-cut winner to me is a journal article published in a reputable journal, because it has to be peer reviewed before publication. After that, I really don’t know… is it the book/conference paper/report/web article/chapter/newspaper article? My answer is it depends. I am not sure if this exercise was supposed to be such an intense thinking exercise, but it has been for me. Since I have to choose, I am going to go with a book.
As for the bottom two resources, I think the blog and the web article are going to be lowest, but again… it depends. Is it the blog of an acclaimed educational thinker? Is the web article published on a world-renowned educational site? Then these would move up the list for me. But, because ensuring objectivity, is the number one thing to me, and because anybody can start a blog or publish to a website (some websites), I will have to rank these two as my lowest.
I, too, seem to be having trouble with disappearing posts. The link to my google doc is https://docs.google.com/document/d/1QO6t_PjmjR0eUn64EOYvWicllZyuT8zluCnj1x9kzDs/edit?usp=sharing
ReplyDelete(apologies for the late post) My top two would be journals (peer-reviewed) and chapters. Bottom two would be blog entry and newspaper/magazine article. Being so late to the party I'm not sure how much I could contribute to what has already been said without repeating much of it but I did come across some interesting notes on evaluating some sources. For scholarly journals I found a resource that suggested using the JournalGuide website and I was wondering if Robin or anyone had any experience with it. The link is https://www.journalguide.com. As well, it was suggested on another site that you can use Google Scholar Metrics (https://scholar.google.com/citations?view_op=top_venues&hl=en) to find the top journals in various disciplines. I'd be interested to know if experienced scholar agree with the rankings which seem to be based on citation statistics from Google Scholar.
ReplyDelete-Phil
This is helpful Phil - you might also consder ranking of education and technology journals at http://patricklowenthal.com/2015/12/ranking-educational-technology-journals/
ReplyDelete