About Research 2

In today's great flood of media items on scientific topics including the environment, not only journalists but also governments, interest groups, general writers and websites (including Ozcranes!) quote science to support their claims or policies. Are they using the most reliable sources, and quoting them fairly?

Some stories, claims or even major policy will turn out to be citing anecdotes: interesting, but not research (Research Tour Part 1). Some do quote research studies, though. Here in Part 2 of the Tour, we compare the places these research studies are published: talks at conferences, some government or NGO reports, scientific books, scientific journals, and more. How do ‘peer-review’ and ‘grey literature’ relate to reliability?

What is..

..‘Peer Review’

A kind of scientific quality control where scientists open up their work for scrutiny by other experts, before publishing. Independent expert reviewers, mostly volunteers, are known as ‘referees’: so a ‘refereed journal’ is the same as ‘peer-reviewed’.

This can be a pretty gruelling process. Think of Carroll D. Littlefield (our scientist from the Abstract in Part 1) wading through swamps year after year, tracking broken eggshells and predator scats, dodging perhaps angry crane parents.. then there's the analysis phase, writing up, and submitting for publication. Scientists and research students would certainly empathise with quotes from the ‘Sense About Science’ (UK) peer-review Working Party –

Few outside the scientific world can be expected to know about the immense effort, and even pain, that is experienced before research is ready to be published as a peer-reviewed paper in a scientific journal… Scientists submit their research findings to a journal, which sends them out to be assessed for competence, significance and originality, by independent qualified experts who are researching and publishing work in the same field (peers). This is known as ‘peer review’. Despite its extensive use and recognition among scientists in assessing the plausibility of research claims, in the rest of society very little is known about the existence of the peer-review process or what it involves.

Is peer-review the same as editing? No, most scientific books and conference proceedings are not peer-reviewed. They have editors, sometimes with major input, but this is in-house vetting, not external and independent. The same applies to supervision of academic dissertations (theses).

There are over 11,000 peer-reviewed science journals, and numbers increase every year. The most influential are the 75% or so that are ‘indexed’ (make up the scientific article databases). Size doesn't matter – peer-reviewed, indexed journals may be small or large, fortnightly or annual, commercial or fully voluntary. Citations are also used to indicate prestige.

The whole process of peer review is periodically debated in the scientific community, the prestigious weekly science journal Nature has a recent article (free access) on the fascinating history of peer review and current debates on its future. Elsevier and other major journal publishers have a Peer Review page covering many issues surrounding its crucial role.

top TOP

..‘Grey literature’

Grey is about universal availability. Academic journals are not grey literature. Is this because they are peer-reviewed? No, only some scientific books are peer-reviewed but most libraries will define them as non-grey. Is ‘grey’ low quality? Not necessarily at all, eg a scientific journal article will be just one fraction of a (grey) dissertation or (grey) technical report that holds much extra material. Some grey literature is prestigious.

Types of scientific research literature

Science journal 11,200 (76% indexed) No
Science Book A few No
Conference talk or program No Yes
Conference Proceedings A few Yes
Dissertation or thesis No Yes
Government report No Yes
Interest Group report No Yes
Technical paper No Yes

Availability Scientific journals and books are marketed universally (albeit sometimes not-for-profit) and anyone with a (very) good budget or library, can subscribe to, or read any science journal or book. Also in theory, journals keep their archives for ever, giving long-term continuity. Government agency reports however may only be available on special order over a limited time period, or may be confidential and never released publicly despite containing important data and conclusions. Conference Proceedings (papers collected into a volume after a conference) are often hard to obtain, but crane research is fortunate that crane Conference Proceedings are available online through the International Crane Foundation Library.

The limited availability of ‘grey’ publications makes writers wary of citing (quoting) them in support of their research unless absolutely necessary, as readers may have trouble tracking down the source to query or extend the research. But (non-grey) out-of-print books can be equally hard to find!

top TOP

Crane research

Common Crane

Literature types aren't mutually exclusive, there's a natural progression through the life of a research project. A PhD student studying cranes might give a talk to a conference with an abstract in the program; the final dissertation may be up on the university website, or the International Crane Foundation Digital Library; key findings of the research may be written up for scientific journals (abstracts usually available free on the web).

← Common or Eurasian Crane, Grus grus (Ian Montgomery)

The researcher may contribute a chapter to an edited scientific book; then might be hired to do a report on cranes for a government agency or wetland conservation organisation. All are very valuable resources, but it's fair to say that articles in peer-reviewed, indexed journals cited by later authors, will have the most widespread circulation and scientific standing.

top TOP

Checking source quality

You can apply the Research Tour principles anywhere people are quoting science to support their statements – the media, websites including Ozcranes, government reports and NGO documents. Are claims based on studies using scientific methods; have the principal findings of these studies appeared in peer-reviewed publications (which); do citations in the studies really support their arguments; finally is the media (or whoever) fairly quoting the studies, or stretching them beyond the original?

Of course governments must govern and make decisions every day, and lobby organisations have their campaigns, and so on. But you may be surprised how much policy, legislation and opinion on scientific topics is not backed by science, although some might be, in time.

«Back to Research Tour Part 1

Change AT to @ and DOT to . in email addresses