Final Report Summary - OBJECTIVE SCIENCE (Quantifying objectivity in the natural and social sciences)
Every year, over 1.4 million scientific articles are published worldwide. How many of them are objective and unbiased descriptions of real phenomena? This crucial question inspired the project, which aimed to collect survey data on misconduct (objective 1), compare the amount of publication bias between fields (objective 2), and assess whether and how the popularisation of science could further bias our scientific understanding of the world (objective 3). These objectives have all been met, in some cases by developing new approaches and techniques.
Objective 1) The first systematic review and meta-analysis of survey data on misconduct was conducted.
Results and conclusions: On average, across disciplines and countries, two percent of scientists admitted to have 'fabricated' (made up), 'falsified' or 'altered' data at least once, and up to 34 percent admitted other questionable research practices including 'failing to present data that contradict one's own previous research' and 'dropping observations or data points from analyses based on a gut feeling'.
Objective 2) Several techniques to compare publication bias were tested. The most versatile and effective consisted in sampling papers that claimed to have 'tested a hypothesis' and then verifying whether in each paper the authors concluded to have found a positive result (full or partial support for the tested hypothesis) or a negative one.
Results and conclusions:
Four papers (two still in preparation) will report this part of the project, as detailed below.
2a) An analysis of 2434 papers in all disciplines revealed that studies in the physical sciences are much more likely to report 'negative' results than studies in the biological and the social sciences. This supports the long-standing philosophical hypothesis, dating back to Auguste Comte (1798-1857), that disciplines can be ranked according to their level of rigour and objectivity.
2b) A subsample of the above papers revealed a paradoxical pattern: more positive results are published in the United States in which, according to data from the National Science Foundation (NSF), academics produce more papers per capita, independent of discipline and amount of research funding. These results support the hypothesis that competitive academic environments increase not only scientists' productivity but also their bias against 'negative' results.
2c) This analysis assessed the factors that increased citation rates of papers across disciplines. After controlling for discipline and other confounding factors, papers received more citations when they reported a support for the tested hypothesis (in preparation).
2d) Analysis of 1158 papers published in 2008 and 2009 in all disciplines replicated the results obtained in the analyses above, and compared directly the levels of bias between the United Kingdom, United States, China, Japan, South Korea, Singapore and Taiwan (in preparation).
Objective 3) By analysing 150 popular science articles published in 2008 and 2009, this study identified factors that make a research finding newsworthy in major Italian and British newspapers, and compared the accuracy of scientific communication.
Results and conclusions: Papers accompanied by a press release, and papers in 'soft' disciplines were significantly more likely to make the news, both in Italian and British newspapers. However, the major determinant of newsworthiness was not scientific but merely nationalistic: Italian newspapers reported more research done in Italy than in the UK, and vice versa. The style and level of detail in reporting science was similar in the two countries.
General impact and use: The outcomes of the project have made significant contributions to understanding of the causes and prevalence of scientific bias and misconduct, phenomena that are of interest not only to academics but also to policy makers, journalists and the general public. The scientific and media impact obtained by the publications is witness to the relevance of these results.
Objective 1) The first systematic review and meta-analysis of survey data on misconduct was conducted.
Results and conclusions: On average, across disciplines and countries, two percent of scientists admitted to have 'fabricated' (made up), 'falsified' or 'altered' data at least once, and up to 34 percent admitted other questionable research practices including 'failing to present data that contradict one's own previous research' and 'dropping observations or data points from analyses based on a gut feeling'.
Objective 2) Several techniques to compare publication bias were tested. The most versatile and effective consisted in sampling papers that claimed to have 'tested a hypothesis' and then verifying whether in each paper the authors concluded to have found a positive result (full or partial support for the tested hypothesis) or a negative one.
Results and conclusions:
Four papers (two still in preparation) will report this part of the project, as detailed below.
2a) An analysis of 2434 papers in all disciplines revealed that studies in the physical sciences are much more likely to report 'negative' results than studies in the biological and the social sciences. This supports the long-standing philosophical hypothesis, dating back to Auguste Comte (1798-1857), that disciplines can be ranked according to their level of rigour and objectivity.
2b) A subsample of the above papers revealed a paradoxical pattern: more positive results are published in the United States in which, according to data from the National Science Foundation (NSF), academics produce more papers per capita, independent of discipline and amount of research funding. These results support the hypothesis that competitive academic environments increase not only scientists' productivity but also their bias against 'negative' results.
2c) This analysis assessed the factors that increased citation rates of papers across disciplines. After controlling for discipline and other confounding factors, papers received more citations when they reported a support for the tested hypothesis (in preparation).
2d) Analysis of 1158 papers published in 2008 and 2009 in all disciplines replicated the results obtained in the analyses above, and compared directly the levels of bias between the United Kingdom, United States, China, Japan, South Korea, Singapore and Taiwan (in preparation).
Objective 3) By analysing 150 popular science articles published in 2008 and 2009, this study identified factors that make a research finding newsworthy in major Italian and British newspapers, and compared the accuracy of scientific communication.
Results and conclusions: Papers accompanied by a press release, and papers in 'soft' disciplines were significantly more likely to make the news, both in Italian and British newspapers. However, the major determinant of newsworthiness was not scientific but merely nationalistic: Italian newspapers reported more research done in Italy than in the UK, and vice versa. The style and level of detail in reporting science was similar in the two countries.
General impact and use: The outcomes of the project have made significant contributions to understanding of the causes and prevalence of scientific bias and misconduct, phenomena that are of interest not only to academics but also to policy makers, journalists and the general public. The scientific and media impact obtained by the publications is witness to the relevance of these results.