Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Evaluating sources: Find out more

How to determine the relevance of sources?

klik voor amker van deze animatieTo determine if a piece is relevant, you may try to answer the following questions:

  1. Does the source help you to answer your main questions and sub-questions?
  2. Does the source answer your whole question/sub-question or only one aspect?
  3. To what extent does the main question of the source you found match with your own questions?
  4. How strong are the similiarities between the research object or the analysis unit in the piece you found and those in your own paper/thesis? The research object may be a period, or a person, a group, an area, a substance, a disease, a proces etc.
  5. Is the context of the research object the same as in your case?
  6. When was the piece published and when was the research written about executed?

Think that you will rarely find a source that provides a complete answer to your main questions and sub-questions and that gives a report of the exact same research or problem you are working on. Thank goodness, otherwise your work would have no use at all....

How to determine the scholarly nature of sources?

The crux of science lies in the extent to which an author/researcher performs his work objectively and makes it verifiable. In determining the quality and scientific nature of sources you may start from three kinds of checks:

  1. Check by others, preceding publication
    • editors: editors of scientific journals are stricter than editors of non-scientific journals
    • publishers: some publishers only publish scientific books
    • peer review: some journals but also some book publishers ask experts for a (blind) judgement before publication
    • search engine/online bibliography: some search engines only include articles from high-quality, peer reviewed journals (for instance Scopus and Web of Science
    • financiers: some journals demand to know who funded the research
  2. Check by others, after publication
    • reviews (in the case of books): are the reviews positive?
    • citations (mainly in the case of articles): is the piece cited often (taking into account how long it has been available) and more importantly:what is being said?
  3. Check by yourself
    • are author and date of the text mentioned (particularly in the case of webpages)?
    • affiliation of the author: the author's job may tell you more about the scientific level, for instance if the author is employed by a good university
    • is the target group mentioned: (in particular in the case of websites and reports)
    • presence of explicit research questions and conclusions
    • presence of an explanation of the method used: how was the research conducted, where are the data coming from?
    • presence of enough and high-quality literature references or notes: what insights are used?
    • language level and well organised text

Database/search engine choice as a first crude filter of sources

Citation databases such as Scopus and Wen of Science (image) are highly selectiveThe selection criteria of the database or search engine you use can give you more certainty about the basic quality of the sources you found. But what criteria do the makers of these databases use?

Citation databases: strictly scientific/scholarly

A very strict database is Web of Science. The selection proces of new journals at Web of Science is partly based on citation data: are there many references to the new journal in the journals already included in the database? In addition the editorial quality and the fixed publication frequency is an important criterion. The inclusion criteria of Scopus are also strict, but  the selection relies more on the the advice of a 'content selection & advisory board' than on citation data. For instance Scopus looks at the geographical spread of editors and authors of a journal. Both Web of Science and Scopus make peer review a condition that must be met. With these databases getting included may take a journal several years..

Subject related databases: scientific/scholarly

Subject related databases (for instance Business Source Premier, (Ebsco) Hospitality and tourism Complete (Ebsco), Lieusre and Tourism Databsae (CABI) try to make searchable as many subjects as possible in a discipine. Almost all of them include articles, but there is an enormous difference among the databases when it comes to the inclusion of books, reports, conference proceedings or MA theses. A (small) part of the sources you will find here will not be peer reviewed.

Google Scholar: scientific/scholarly but with 'pollution'

Google Scholar's selection criteria are fairly broad and are partly technical by nature. Google Scholar hardly offers anything to go on as far as scientific quality is concerned. It is true that much material from strictly scientific journals can be found here, but Google Scholar also automatically recognises texts on the web which are probably not scientific, such as articles published on the website of the author or on a university website. That is why you will also find non-peer reviewed material, concept versions, library guides, students' papers etc. Also books from Google Books are added automatically which are recognised as scientific.

Library catalogue: sources relevant to your courses

In library catalogue you will find a wide range of material. Here the main selection criterion is whether something is relevant to education and research of an academy. So a library catalogue may also contain popular-scientific material and also non-scientific sources. In addition to the catalogue BUAS has a broader Metasearch.

Meta databases and meta catalogues: all kinds and levels

Large hybrid search systems such as BUAS' Metasearch, the Dutch Picarta and worldwide WorldCat offer the less to go on due to the many kinds of sources and databases from which they combine all kinds of material. In particular at Picarta and WorldCat because all sources of public libraries are indirectly included there is absolutely no certainty about the scientific level.

The journal as criterion

Nature wijst >90% artikelen afhttp://webform1.nhtv.nl/mediatheek/Link.aspx?ID=2999You can derive the scholarly nature (but not automatically also the quality!) of journal articles from the journals in which they are published. If all is well, you will find the applied criteria in the journal itself or on the website of the journal. Many scientific journals check these via the so-called peer review process. To this end, the article is blindly presented to experts invited by the editors (may be worldwide). In this way the journal guarantees a kind of minimal scientific quality. But it does not mean that articles from peer reviewed journals are always better thanarticles from journals without peer review.

If a (modern) journal is included in Web of Science or Scopus, you may expect that it is peer reviewed. In all other cases you will have to check it yourself on the site of the journal or in the journal list.

Sometimes journals make high demands on articles and some of them reject more than 90 percent of the submitted articles, a large part already before the process of peer review. This rejection rate may even give a journal more status.

Demands a journal makes are always with regard to originality and the interest to the (continuation of) science. In addition there are many extra requirements, such as a clear argument, language and the right references. Sometimes it is demanded that data are made public.

But be careful: also seriously scientific journals contain non-scientific parts such as letters to the editor, book reviews, news, comments, opinion pages and editorials. These parts are often exluded from the peer review check. As a result, you will have to treat them differently when you use them.

Types of journals

Although the subjects written about can be the same in all periodicals (climate change, the financial crisis, popular music) there are considerable differences between journals. You use scientific journals to find out if your subject has already been researched and to cite from. Professional journals are also used to make yourself familiar with your discipline and its organisation, and sometimes also to cite from. General news magazines provide information about social, economic and political issues. Popular journals mainly focus on entertainment.

 

                           Scientific journal Professional journal Opinion magazine Popular journal
Authors

Always mentioned, always scholars/scientists,

usually with work address, editors only take care of introduction

Usually  mentioned, usually scholars/scientists, editors sometimes take care of introduction Usually mentioned, sometimes anonymous, many articles written by the editorial staff Usually mentioned, sometimes anonymous, very many articles written by the editorial staff
Payment Editors and authors work for free, editorial board paid professional(s) core editors are paid, authors work for free editors are paid, articles by others are also paid for core editors are paid, articles by freelancers are also paid for
Sources Almost always literature references > 10 Sometimes literature references < 5 usually no literature references usually no literature references
Quality check Peer review Editors Editors Editors
Type of content Research results, analysis, discussion, lots of jargon News, trends in the professional field, shorter analytical pieces News, interviews, background, discussion, politics reports, pictorial articles
Typical length of article 2.500-10.000 words 400-2.000 words 500-1.500 words 300-1.200 words
Page numbering Continues throughout the volume sometimes continuous, usually for each separate issue for each separate issue for each separate issue
Design Simple cover, few pictures, many diagrams, tables, hardly advertisements, many rules about the structure and layout of the article Relatively cheap design, relatively many advertisements many news pictures and pictures of persons, many advertisements colourful, many pictures

Typical publication frequency

4-10/y

8-12/y

26-52/y

12-52/y

Examples

PLOS One, Environment & Planning. Tourism Management, Annals of Tourism Research American Economic Review

Economisch Statistische Berichten, Travel Magazine, Geografie

Economist, Groene Amsterdammer, Elsevier, Vrij Nederland, Opzij, Time, Der Spiegel

National Geographic, Kijk, Avantgarde, Avenue, Elle, Quest, OOR

Some professional journals focus on bringing a discipline to the attention of a large audience and as a result share many characteristics of popular journals. Examples: Historisch Nieuwsblad, Filosofie Magazine, Psychologie Magazine.

Peer review: the crux of scientific communication?

some publishers (e.g. Copernicus) experiment with new forms of peer reviewAccording to many, peer review, the system by which fellow experts judge the quality of a piece before publication is one of the pillars of science. Editors can never have enough expertise to judge submitted articles or books. Experts can. Without them knowing the identity of the author, they assess whether the piece meets the scientific norms: stating where the data originate from, a logical analysis, references to relevant sources. Besides they say what  the article has to contribute to science. Peer reviewing an article is more about the question of a piece being written according to a good scientific method than about reviewers agreeing with the content. The advice to the editors will be either publishable, publishable with certain adjustments/ not publishable.

The peer rewiew system is subject to much criticism, although many people say that is the 'best worst system' we  have. Criticism focusses on:

  • Delays: the system can delay publication for months, sometimes for over a year
  • Interests: sometimes the impartiality of reviewers is doubted: in small disciplines or specialisms it is often easy to find out who the author is. As a result, publicatons by others may be prevented or the reviewer, acting out of self-interest, may demand that the author refers to the articles of the reviewer
  • Some reviewers who are in great demand spend a lot of (unpaid) time on peer reviewing articles
  • Strongly innovative, experimantal research and the accompanying publications have less chance to be published because they deviate from the norm and will be deemed less publishable by reviewers
  • The English has to meet high standards and may cause problems for  non-native speakers. As a result, these demands may prevent international publication of research that may well be excellent.
  • Stating the interest for science is arbitrary

There are sites where authors share their experiences with the peer review of journals. The Dutch initiative SciRev is one of them.

Some journals, such as PLOS One (Public Library of Science One) are experimenting with new forms of peer review. Peer reviewers only indicate if the articles are written in the right method. Assessing the scientific interest is done at a later stage, by the readers. The use of this method leads to a speedier publication of many articles.

Disciplines manage sources and argumentation in different ways

Publication cultures of disciplines differ:

  • The ratio between used publication forms: in Humanities books are very important, in Natural and Biomedical Science the journal article is the main form of scientific communication. Social Sciences are between the two. In all disciplines the importance of journal articles increases. In the Humanities diaries and archive records are often used as primary source. In Social Sciences statistics and interviews are commonly used primary sources.
  • In the Biomedical Sciences peer review is of the utmost importance. In some other disciplines articles are often quickly presented to the outside world as working papers or pre-prints and only at a later stage are they published 'officially' in a journal.
  • Nature artikel met honderden auteursThe number of authors that have worked on the article is much higher in the Medical and Natural Sciences than in the Social Sciences and in Humanities. In extreme cases the article carries  more than a thousand names of authors
  • In some disciplines (for instance History) literature is referred to in the form of footnotes, others use the author-date system with the name of the author and the date in the text and others again (for instance Physics and Astronomy) only use numbers to refer to literature.
  • In Social Sciences and Humanities the structure of journal articles varies more than in the Medical and Natural Sciences, that use a more rigid structure.
  • In  Humanities (for instance Philosophy) and less so in Social Sciences, the argumentation focusses more on convincing the readers and presenting plausible arguments  than on hard proof based on empirical research, mainly because many subjects are not suitable for this kind of approach

Reports: scientific or not?

Reports are often heavily debatedIn many disciplines reports are a rich, up-to-date and readable source containing concrete information and many source references. They are producec by authors belonging to organisations or by guest authors (sometimes scientists) which have been invited. The publishing or paying organisations may be:

  • Ministries
  • Semi-governmental organisations (for instance health care institutions and their umbrella organisations)
  • Governmental think tanks (in the Netherlands WRR, SCP, Centraal Planbureau, Plabureau voor de Leefomgeving, Raad voor de verkeerrsveiligheid etc.)
  • Governmental scientific institutions (NIOD, NIOZ, KNAW etc.)
  • International governmental organisations (United Nations, European Union etc.)
  • Statistical bureaus (CBS)
  • Commercial think tanks (Clingendael, Brookings, Pew etc.)
  • Companies
  • Professional associations (Nederlands Instiuut voor Psychologen, KNMG etc.)
  • Branche organisations and umbrella organisations
  • Non governmental organizations (NGO's, e.g. Greenpeace, WWF, Human Rights Watch etc.)
  • Pressure groups
  • Groups of scientists (e.g. IPCC climate panel)

The level of the reports published by these organisations can be very high, and sometimes even scientific standards are used. In almost all cases, however, there are parties involved who commissioned the publication and who sometimes determine which outcomes are presented to the outside world and which are not.

Critical reading

Even if you do all the checks on these pages there are no guarantees that the pieces you will be reading are true, objective and unbiased. You will always have to remain careful and above all, read critically. Make a sport out of spotting the bias in articles, books and reports.

Evaluating websites

If you want to use information from websites other than regular scholarly articles and books you should be extra careful and think about the role you give that information in your argument or analysis.

Ask yourself the following questions when reading webpages and be extra careful if the answer is negative most of the times.

  1. Is the name of the author/maker available (and do you know more about the author)?
  2. Can you find an e-mailaddress of the author/maker?
  3. Is the webpage free of (lots of) advertisements?
  4. Is the use of language careful, not childish, correct?
  5. Is it clear how the information on the page came about?
  6. Are sources mentioned (so no phrases like "research shows that") without any quotation of sources?
  7. Are claims well-founded (so no phrases like "as is common knowledge")?
  8. Is it a balanced piece or does all information point in the same direction?
  9. Is the author open about matters still unknown or uncertain?
  10. Can you find when the page was written or updated?
  11. Is the page unbiased or in any case without strongly political or commercial aims?

These questions do not relate of course to the use of scientific articles or e-books published on websites.

Lingo

  • peer review = blind check on scientific quality by fellow experts
  • spoof = fake publication, usually satirical

Good use of non-scientific sources

Whether you can use non-scholarly sources depends on:

  • your discipline
  • the sort of work you are writing
  • the role the source plays in your argumentation

You can use non-scholarly sources as:

  • object of your research (for instance in what way is a subject discussed in the popular media)
  • temporary solution if no scientific research is available yet
  • primary source (archival records, letters, interviews, statistics, news)
  • indication of social relevance
  • illustration of your point

Important scientific publishers

There are many hundreds of scholarly publishers. In addition to the large ones you will find below, there are many small ones in all language regions and there are also many universities that have their own scientific publisher (usually called ... University Press)

  • Academic Press
  • American Chemical Society
  • American Geophysical Union
  • American Physical Society
  • Armand Colin
  • Ashgate
  • Bellwether
  • Biomed Central
  • Blackwell
  • Bohn Stafleu
  • Boom
  • Brill
  • Cambridge University Press
  • CRC press
  • De Gruyter
  • Earthscan
  • Edward Elgar
  • Elsevier
  • Emerald
  • Franz Steiner Verlag
  • Gale
  • Hachette
  • Harmattan
  • Institute of Physics
  • Karger
  • Kluwer
  • MIT Press
  • Nature(MacMillan)
  • Oxford University Press
  • Palgrave
  • Routledge
  • Sage
  • Springer
  • Taylor & Francis
  • Thela
  • Wiley

Book reviews

Content

  • Descriptions and reviews of (new) books. The ratio between description and review varies widely. In literature (fiction) which author/journalist has written the review is an important factor. So a book review is by no means an objective piece

Use reviews to:

  • Find out which new books are thought important
  • Test your own opinions against those of others
  • Take note of the importance and, in a very limited way, the content of a book without reading it yourself

To be found in:

  • Newspapers
  • Opinion magazines
  • Literary journals
  • Scientific journals
  • Professional journals

some journals (e.g. The New York Review of Books) specialise in books reviews

 

 

 

 

 

 

 

 

 

Search with:

  • subject related bibliographic databases (scientific journals) 
  • Web of Science (scientific/scholarly  journals )
  • LexisNexis (newspapers, opinion magazines)
  • Google Scholar (scientific/scholarly journals)
  • Google
  • Google Books (reviews by specialised sites and by readers)
  • Amazon (reviews by specialised sites and by readers)

 

Spoofs: who gets the joke?

Sometimes entire publications are imitated having as aim satire, criticism, identity damage or commercial gain. That we are dealing with a fake publication may be clear from the start, but it may be done so subtly that you hardly notice it. In that case the exposing of a spoof is a question of experience and reading and viewing very carefully

NRCNext spoof

NRC Next became victim of a spoof in the build-up to the 2012 elections of the Dutch parliament