Overcome failure to Publish nEgative fiNdings

Final Report Summary - OPEN (Overcome failure to Publish nEgative fiNdings)

Executive Summary:

Publication bias, or as has been suggested more recently, dissemination bias, results from a tendency of researchers to submit - and of journals to accept - journal articles of research based on the nature, direction or strength of findings, resulting in a skewed available evidence base for medical decision making. Thus, health care practitioners, public health professionals and policy makers cannot rely on the entire, unbiased evidence from clinical research.

To tackle this problem, the EU has granted funding to the OPEN project (To Overcome failure to Publish nEgative fiNdings). OPEN is a 24-month project, running from November 2011 to October 2013, aiming to develop evidence-informed recommendations, which focus on reducing dissemination bias.

To assess the current evidence of dissemination bias, OPEN has been conducting a series of systematic reviews with regard to the non-publication of research findings as journal articles and the resulting dissemination bias. These reviews address aspects such as existing terminology to describe problems of publication and related biases, available methods to detect and measure dissemination bias and the extent – and impact of – the problem of non-publication of research findings as journal articles. In addition, OPEN has sought to assess and evaluate the policies and procedures in place for preventing dissemination bias by the main parties involved in funding, conducting and publishing clinical research: funding agencies, the (pharmaceutical) industry, research ethics committees, research institutions, researchers, trial registers, biomedical journals, regulatory agencies, and benefit assessment agencies. To this aim, these key groups have been surveyed and available documents pertaining to this question have been analyzed.

Subsequently, the findings of OPEN have informed a two day recommendations workshop of OPEN project partners as well as selected key stakeholders from across the world in May 2013, which aimed at developing and refining a set of targeted recommendations that are designed to specifically consider the roles that the respective stakeholder groups should play in reducing the incomplete registration and dissemination of research findings.

Overall, 47 recommendations tailored specifically to the different key groups have been jointly agreed upon, as well as 4 general recommendations which apply to all key groups. These recommendations are currently being disseminated by means of multiple publications, as well as presentations at conferences and further measures such as our website, blog entries and coverage in non-specialist media such as newspapers.

Through dissemination of our results and our consented recommendations, we are confident to achieve more awareness of the problem of dissemination bias and thus effectuate a change in practice, leading to less dissemination bias. Since dissemination bias can inflate apparent drug efficacy and thus make it difficult to distinguish effective from ineffective drugs, and can conceal drug harms, patients will benefit from a more comprehensive evidence base by experiencing more benefit and less harm from treatments.
Project Context and Objectives:
Project context

Full information about completed and on-going clinical trials is the indispensable base for decision making about medical therapies and diagnostic procedures by patients and doctors. Equally, researchers and research organisations, ethics boards, governments and health system agencies, courts for social justice, pharmaceutical companies and all professional groups of the health care system are dependent on unbiased information.

However, authors tend to submit - and journals tend to accept - articles based on the nature, direction or strength of findings (for approximately 50% of all launched trials, results are never published as a journal article). Hence, this non-publication of research findings leads to a skewed available evidence base for medical decision making (dissemination bias).

Dissemination bias can inflate apparent drug efficacy and thus make it difficult to distinguish effective from ineffective drugs, and can conceal drug harms. Therefore, this non-comprehensive evidence base may result in in patient harm and less than optimal patient benefit.
Dissemination bias also implies a waste of resources and a considerable investment by funders without any return; in addition, previously conducted but unpublished studies may be replicated, thus requiring double financial investment. It is also likely that some reimbursement decisions by national HTA institutes are currently based on biased systematic reviews so that societies continue to pay for ineffective treatments.

Furthermore, failure to fully publish the results of clinical research is unscientific and unethical practice – patients who have consented to participate in a trial, because they are led to believe that such research is necessary for the advancement of medical science, should be respected for their personal contribution. However, if results of clinical trials do not get published, their contribution is for nought.
So why has this problem not yet been tackled? A first answer would be a lack of awareness of the problems dissemination bias poses, resulting in a lack of perceived necessity of change. Another reason is complexity of the system, giving ample space for all stakeholders to avoid appropriate changes and draw attention to deficiencies outside their responsibility.

Objectives

The first objective of OPEN was to identify and explore the current evidence on dissemination bias. Therefore, systematic reviews to assess 1) the current state of the literature addressing the definitions and the nature of dissemination bias, 2) approaches of detecting dissemination bias in systematic reviews and approaches of measuring dissemination bias, and 3) the extent and impact of dissemination bias on systematic reviews and health technology assessments were conducted.

The second objective of OPEN was to describe current practices by various key groups involved in knowledge generation and translation to thus provide insights on how to avoid or reduce dissemination bias and identify ways to change this practice. Also, attitudes towards dissemination bias were investigated. This was addressed by surveying the main parties involved in funding, conducting and publishing clinical research: funding agencies, the (pharmaceutical) industry, research ethics committees, research institutions, researchers, trial registers, biomedical journals, regulatory agencies, and benefit assessment agencies. These results were collated to describe the status quo.
Based on those insights, OPEN’s third objective was to derive concrete recommendations aimed at specific key groups of the knowledge translation and generation process to reduce dissemination bias.

By disseminating our results, we hope to raise awareness of the existence of dissemination bias and the disastrous consequences arising from it, and thus promoting a climate of change. We are confident that the uptake of our recommendations will foster the provision of relevant results from clinical trials to citizens and organisations in the EU, leading to a more comprehensive evidence base and ultimately more patient benefit and less patient harm.
Project Results:
In this section, the main scientific results are briefly described. For more detailed results, please refer to the Work Package Report of each WP. Throughout this document, we will use the term ‘dissemination bias’ rather than ‘publication bias’ as has been suggested by Bassler et al. (see 4.1.3.1).

4.1.3.1 Current knowledge on publication bias, its extent and its impact
Work package 1 aimed to assess the current knowledge on dissemination bias, its extent and its impact by conducting systematic reviews of the relevant methodological literature. The main results of these systematic reviews are given below:
In the existing literature, dissemination bias has been mainly described as selective publication (45/50). A new framework devised by the OPEN project chooses the term “dissemination bias” instead of “publication bias” to incorporate all the associated forms of bias and to reflect the fact that dissemination of results can also be achieved by means other than a journal publication.
To detect, quantify or adjust for dissemination bias, we primarily identified methods based on funnel plots (48/135) with our literature search. However, several (11/135) other statistical methods are also described.
The proportion of studies published that either:
a. received research ethics committee approval (n=16 methodological research projects) or;
b. were registered in trial registries (n=11 methodological research projects) or;
c. were presented as abstracts at conferences (n=264 methodological research projects)
was 45.0%, 49.5% and 39.5%, respectively.
Certain study characteristics were associated with a higher likelihood of publication as a journal article, e.g. significant results or RCT design or having received funding. Thus, it is safe to conclude that non-publication of clinical trial results is not a random process.
On the basis of all data available at this time, no valid conclusions can be drawn regarding the impact of studies that are not published or published in the grey literature on pooled effect measures in meta-analyses. However, based on very limited available data it seems that the over-estimation in effect seen in the cohorts of meta-analyses included in our systematic review is only moderate.

4.1.3.2 Funding agencies of clinical research
Work package 2 aimed to evaluate the policies and procedures of funding agencies of clinical research to prevent dissemination bias, as well as their opinion and views on this phenomenon. In addition, it sought to poll these agencies on the feasibility of different procedures already in place to control dissemination bias. In order to achieve this objective, an electronic survey was conducted among funding agencies from 34 European countries.
68% of funding agencies participating in our survey (n=56) currently have a policy to encourage grantees to publish the results of the work for which they had been funded (in 2005: 59%). About two thirds of agencies follow up with grantees to verify that they have published the results of the research for which they were funded, although less than half do so actively (e.g. during evaluation as opposed to simply asking grantees to send their publications once they are ready). 24% do not follow up with grantees at all on this matter. However, up to 77% of surveyed agencies did not know in detail whether research funded in 2005 had been published. These numbers show that despite having policies that encourage publication of results, most agencies have difficulties implementing such measures and following up with the appropriate parties to ensure compliance.
About 50% of funding agencies participating in our survey view dissemination bias as “not a problem at all” (14.3%) or “somewhat a problem” (35.7%), with the remaining half seeing it as “a problem” (32.1%), “a serious problem” (14.3%), or “a very serious problem” (3.6%). 29% of participants in this work package´s survey believed that researchers should be the ones deciding which research results to publish.
When asked to provide initiatives and/or ideas to control dissemination bias, only three agencies provided answers, which were: “mandatory registration of trials”, “retraction watch blog” and “funding agencies should be more careful”. Furthermore, only 23% of respondents agreed that policies aimed at reducing dissemination bias are realistic and feasible.

4.1.3.3 Pharmaceutical industry
The objective of work package 3 was to evaluate policies and procedures used within companies to prevent dissemination bias and to assess experiences of dissemination bias when trying to get company research published in medical journals. To this aim, a survey was sent to 24 pharmaceutical companies and 6 devices companies.
Overall, 12 companies completed the survey of which 11 were pharmaceutical (7 top 10 companies) and 1 non-pharmaceutical. Questions about policies and/or procedures concerning publication of research results were divided into three blocks separately for phase 2, phase 3, and phase 4 studies.
Most pharmaceutical companies (81.8%, 77.8%, 77.8%) participating in our survey (n=12) claim to have a policy regarding publication of phase 2, phase 3, and phase 4 studies. However, the majority of this information is not in the public domain (54.5% for phase 2, 71.4% for phase 3 and 4 trials). Non-publication of at least 50% of research work that could be submitted for publication in a journal was acknowledged by a minority (12.5%) of the companies, whereas 87.5% reported that at least 50% of research work was submitted for publication. Reasons given for non-publication varied and included: limited resources, commercial competitive considerations, ceasing to develop the product, and addressing topics that might not be of interest for readers.
Almost half of respondents (44.4%) reported that they have experienced work being rejected by journals for being industry funded. All of these were eventually submitted to another journal.

4.1.3.4 Research ethics committees
Work package 4 conducted a survey of European Research Ethics Committees to find out more about current practices and policies to address the problem of incomplete reporting and publication of trials and their results. Furthermore, the survey recorded attitudes of members of Research Ethics Committee (REC) towards trial registration and dissemination bias.
Overall, 54% of REC respondents (n=199) reported that they regularly check, whether or not trials have been registered appropriately. However, figures varied for different types of clinical trials.
Only a small proportion of respondents reported that their REC had specific procedures in place to enforce the publication of trial results, and only 22% of respondents stated their REC checks that research findings are published in some form – some individual answers stated limited resources and not being responsible as reasons for this.
6% of respondents thought that selective reporting has no negative impact on public health at all, and 6% gave examples for circumstances in which it would be acceptable to selectively report trial results, e.g. “small studies, statistically insignificant results with no negative findings”.
A majority agreed that the work of their REC in this area would benefit from more extensive and comprehensive guidelines and - possibly - legislation to regulate the dissemination of trial results.

4.1.3.5 Research institutions
This work package aimed to evaluate the policies and procedures of research institutions in the biomedical field to prevent dissemination bias, as well as their opinion and views on this phenomenon. In addition, it sought to poll these institutions on the feasibility of different procedures already in place to control dissemination bias. In order to achieve this objective, an electronic survey was conducted among research institutions from 34 European countries.
The survey was sent to 275 research centres and 71 replied (26% response rate). 76% of research institutions responding to our survey have a current policy to encourage publication of results (2005: 51%). In addition, about three quarters of participating institutions try to follow up with researchers to ensure that they adhere to these policies, although only 58% of these do so actively by requesting proof of publication. 24% do not follow up with researchers at all on this matter. However, and in clear contrast with these numbers, only 31% of participating institutions had knowledge of the number of publications related to the research conducted in 2005, while 52% were only able to provide an estimate and 17% did not know at all. These numbers show that despite having policies that encourage publication of results, most institutions have difficulties implementing such measures and following up with the corresponding parties to ensure compliance.
13% of respondents view dissemination as “not a problem at all”, and 24% believe that researchers should decide the research results that will be published and 4% believe that it is the decision of the research institution. 26% believed that both observational and experimental studies should be published only when they have relevant results.
When asked to provide initiatives and/or ideas to control dissemination bias, only nine agencies provided answers, e.g. “easier ‘registration’ of trials”, “Make mandatory to publish a final report regarding all trials that have been started”, “Legal requirements (e.g. FDA Amendments Act; §42b of German drug law) - EMA access to documents policy from 2010 - ongoing discussion at EMA concerning access to clinical trial data”, or “As well as publications being lodged in repositories, it should be common practice for study data to also be deposited for others to analyze (perhaps after a window of time when the researchers have sole access to the data)”. 63% of respondents agreed that policies aimed at reducing dissemination bias are realistic and feasible.

4.1.3.6 Researchers
The aim of work package 6 was to further explore the reasons for dissemination bias and its prevalence among researchers actively involved in performing clinical trials, as well as their opinions and recommendations on how to best detect and prevent dissemination bias. To this aim, we conducted an e-mail survey of different populations of researchers – researchers from trials published in major general medical journals, researchers from trials published in European journals, authors of Cochrane systematic reviews, and general body of researchers (via the OPEN web-site) and also conducted a focus group of clinical researchers and systematic reviewers.
70% of all researchers responding to our survey (n=182) stated that the last trial they had participated in was registered in a public registry before recruitment, and 31.1% had ever posted results of any of their trials in a registry.
More than one third of all researchers had unpublished clinical trial(s), and about a third participated in a trial(s) for which there was/were selective reporting of outcomes and/or analyses. The reasons given for not publishing clinical trial results were predominantly lack of time, perceived unimportance of the results or rejection by a journal.
Only 64% thought that dissemination bias is a serious problem for health care, and 11% were of the opinion that researchers should be the ones to decide whether they want to publish their results.
The respondents were not sure that the current practice of trial registration and posting of trial results in public databases would decrease dissemination bias: 37.0% did not think that these measures are enough and 20.4% were undecided. About half of respondents thought that outlets for publishing clinical trials other than peer-reviewed journals could help to decrease dissemination bias.

4.1.3.7 Trial registries
Work package 7 aimed to assess the quality of registered trial data before and after the implementation of the International Standards for Clinical Trial Registries which have applied to registries in the WHO Registry Network since January 2010 and to determine whether including unpublished data in systematic reviews prevents or actually exacerbates dissemination bias, the reason being that the owners of unpublished, non-peer-reviewed research data may only provide selected data and not the entire data set. The third task seeked to determine whether systematic reviewers use clinical trial registries appropriately to address dissemination bias.
The quality of registration entries in clinical trials registries has significantly improved in several areas since implementation of the International Standards for Clinical Trial Registries. However, further improvements are necessary to increase the usability of clinical trials registers and benefit all those looking for and using information about clinical trials.
Only 9 systematic reviews were identified that included unpublished trials and it was generally not possible to determine which trials within these reviews were the unpublished ones. It was therefore not possible to compare registered data with reported data for these unpublished trials.
Systematic reviewers are not routinely searching prospective registries of clinical trials as part of the process of conducting a systematic review. In the few cases where prospective clinical trial registries are being searched, the details of the searches are very poorly documented and cannot be repeated. This includes lack of reporting of the dates of the search and the search terms used. As a result, it is not possible to determine if systematic reviewers are properly searching clinical trial registries for studies.

4.1.3.8 Journals
Work Package 8 comprised a survey of journal instructions to authors and qualitative research involving interviews with journal editors and publishers to understand reasons for journal policies on trial registration and other measures aimed at reducing dissemination bias.
Of the 200 journals sampled for our survey of journal instructions to authors, only 55/200 journals (28%) required trial registration according to their instructions and a further 3 (2%) encouraged it. The editors and publishers interviewed explained their reluctance to require registration in terms of not wanting to lose out to rival journals, not wanting to reject otherwise sound articles or submissions from developing countries, and because of their perceptions that such policies were not relevant to all journals. Our interview findings also suggest that journal editors may not believe that the benefits of trial registration are that great.
Regarding the role the various key groups in the knowledge translation process play with regards to dissemination bias, the editors and publishers we interviewed recognized that authors, reviewers, funders, research institutions, and editors themselves may contribute to dissemination bias.
For example, some interviewed editors stated that:
• “We don’t publish negative trials”
• “As an editor […] you are going to select the papers that are the most exciting, that are really pushing the field forward […] and in the pre-clinical and basic science area, these are the papers that must demonstrate an effect”
Some blame was given to authors, with points being raised that authors:
• do not submit reports of studies that produce non-significant or unfavourable findings so readily as those with significant or favourable findings;
• exhibit bias in where they send manuscripts so that reports of studies with negative findings (of both types) are more likely to be submitted to lower impact journals;
• try to re-work their data or look at different aspects of it, in order to obtain a positive result.
The policies of researchers’ institutions were highlighted as playing a role in dissemination bias. According to one interviewee: “there are lots of countries where publications have to be published in a journal in the top third of its impact factor category’ to be eligible for inclusion on a CV for a potential employer.” [9]
Perceptions that readers do not want to read negative studies may also explain editors’ decisions.
A number of measures to combat dissemination bias were proposed. These included journals specializing in publishing negative findings, journals selecting submissions on the basis of scientific validity rather than perceived reader interest, publishing findings in databases rather than in journal articles, raising awareness and educating authors, reviewers and editors, as well as trial registration.

4.1.3.9 Regulatory agencies
Work package 9 aimed to evaluate the role of the main regulatory agencies (RAs), including in particular the U.S. Food and Drug Administration (FDA) and the European Medicines Agency (EMA), in controlling failure to publish negative findings from CTs. In this report, results for EMA are presented.
To reduce dissemination bias, RAs have the possibility to follow 5 main routes, each of which helps RAs improve transparency. EMA’s strengths and limitations in relation to these routes are discussed according to EMAS’s current regulations (mainly obtained from a review of the scientific literature).
1. Enforce legislations requiring CT registration and the provision of full protocols (or parts of them) to the public before the CT conduction
EudraCT database collects information on certain interventional CTs for drugs (phase 2-4 CTs conducted in EU or EEA) since 2004. Since 2011, through the EU-CTR website, EMA provides the public with information held in the EudraCT database (in English language). The creation of EU-CTR represents a milestone on the path towards clinical transparency in Europe. Some limitations still exist, including the fact that CTs for surgical procedures, medical devices or psychotherapeutic procedures, as well as CTs on drugs conducted before 2004, are neither registered in EudraCT nor provided in the EU-CTR database.

2. Enforce legislations requiring reporting CT summary results once the study ends
CT result reporting is currently not implemented, but EMA announced a plan to enact policies, which will likely come into force in 2014 and will provide public access to CT results. Though results reporting will likely be required, EMA does not mention in its Commission Guideline (2012/C 302/03) penalties in case of non-adherence to the policy. This may result in a lower compliance with CT result reporting, required by the future regulation.

3. Disclose CT data by providing an online databases of clinical study reports on approved drugs
For each medicine that the EC granted a central marketing authorisation, for the ones that were not authorized, and for those suspended or withdrawn after being approved, EMA publishes European public assessment reports (EPARs), a report, written in agreement with the industry, which are available online and include relevant clinical trial summaries (called Scientific Discussion).
The EPAR database only includes reports for medicines that the EC granted a central marketing authorisation. For instance, drugs approved for the marketing by a specific EU Member State or studies on medical devices do not have reports in the EPAR database. Moreover, most trials conducted after drug approval are not captured and reported in the EPAR database.
Data reported, particularly on phase 3 CTs, is often of poor quality, is incomplete and the corresponding Scientific Discussion has limited methodological details.

4-5. Provision of unpublished regulatory data, upon written request to EMA; collection and provision of raw data / individual patient data
Following the pressure from the scientific community and in response to the complaints by reviewers for failure to obtain CTs unpublished data upon written request, EMA, in 2010, further regulated and widened the access to CT reports and protocols. The new EMA policy now allows reviewers the access to full clinical trial reports of both the published and unpublished studies used for the drug authorization process, whose summaries are included in the Scientific Discussion of EPARs. Limitations of this route include the potentially lengthy delays in receiving a decision or the documents. Also, litigation for denied requests may be both resource and time consuming. Moreover, also after the implementation of the 2010 policy, document requests may be denied for commercial interests and potentially important information may be redacted from documents. Without a formal and clear regulation, it is only up to EMA to decide whether to give or deny access to documents.
EMA is planning to provide access to full CT individual patient datasets to outside investigators. However, no official confirmation is available in the published Commission Guideline.

In summary: although EMA took steps to improve transparency, in particular when it created the EU-CTR website in 2011, it currently has not fully adequate procedures to control dissemination bias, since EMA’s plan to improve transparency has not yet come into effect.

4.1.3.10 Benefit assessment agencies
The objective of work package 10 was to address whether dissemination bias is currently an issue for decision-making instances in the context of pharmaceutical benefit assessment for the purpose of pricing and reimbursement in Europe and if so, how it is being handled and whether it has any impact on the decisions themselves. Both a systematic document analysis and a survey among relevant bodies were performed.
Institutions that are in charge of decision-making on reimbursement and pricing of pharmaceuticals from 36 countries were included in our analysis (i.e. EU member states and candidate countries, EEA and EFTA countries). For the survey, 13 valid responses could be processed corresponding to an 35% of the 37 institutions initially contacted.
Eight out of thirteen countries stated that the institution employs or requires a systematic approach to retrieve information, while respondents from the remaining five countries indicated the use of non-systematic approaches.
While all 13 survey respondents mentioned that their institutions take published data into account for the purpose of benefit assessment, only seven explicitly stated that unpublished information is also considered. The consultation of trial registries was only reported as a prerequisite by one participant, while it is done occasionally in ten others. Manufacturer input is in most cases required to provide lists of all published trials, a practice which is expanded to also include unpublished studies in 4 cases and on-going trials in 2 countries. Only 2 countries always check the evidence retrieved for assessment of risk of dissemination bias.
When asked about future policies that should be applied in order to overcome dissemination bias, most countries (8/12) endorse manufacturers having to sign a statement declaring that all relevant evidence has been submitted. 5 out of 12 countries asked for mandatory standardized posting of full study results to study registers and 4 out of 12 countries stated that public access to European Medicines Agency (EMA) databases could be useful.
As a conclusion, dissemination bias has only rarely been explicitly considered in the process of benefit assessment. However, it is encouraging that only rarely a single source of information is deemed sufficient for benefit assessment processes despite the great variations both in the number of sources used and in the level of transparency of methods.

4.1.3.11 Network meta-analyses
The objectives of work package 11 were to assess the impact of dissemination bias on results of network meta-analyses (NMA), to develop and assess the performance of adjustment methods for dissemination bias in NMA and to develop and assess the performance of methods to detect reporting bias in NMA.
In a particular network of antidepressant drugs, we showed that dissemination bias biased NMA-based estimates of treatments efficacy and modified ranking. Moreover, the reporting bias effect in NMAs may differ from that in classical meta-analyses in that dissemination bias affecting only one drug may affect the ranking of all drugs [1].
We introduced 2 methods to perform sensitivity analysis of NMA results to dissemination bias. The idea behind these adjustment models is to embed bias models into a Bayesian hierarchical model for NMA by adding an assumption that allows reporting biases in different comparisons to be exchangeable across the comparison network. In the same case study of antidepressant drugs, adjustment models showed that NMA of published data was not robust to dissemination bias and provided estimates closer to that of NMA of unpublished data, although not optimal. The validity of such methods depends on the number of trials in the network and the assumption that conventional MAs in the network share a common mean bias mechanism [2].
We proposed a test for dissemination bias in network meta-analysis. It borrows strength from trials across the network under the assumption that biases are exchangeable, that is biases, if present, operate in a similar way in trials across the network. Our simulations showed that the test was fairly powerful and suggest using, for applications, either the result of the largest (most precise) trial or the fixed-effect summary. The former choice may often have a minor advantage. However, it should be used and interpreted wisely. In cases of a statistically significant signal of bias, one should not conclude that this bias is specifically dissemination bias, since it could reflect other practices such as selective analysis and outcome reporting biases. In the absence of significant signals of bias, one should not exclude the possibility of dissemination bias, because a “negative” test result typically does not greatly decrease the prior odds of bias, unless in cases of many trials or substantial between-trial heterogeneity.

4.1.3.12 Final recommendations
Based on the results of the conducted surveys, each work package derived draft recommendations on how to reduce dissemination bias aimed at each specific key group; overall, the findings of the work packages resulted in 70 preliminary recommendations.
To reach a consensus on these draft recommendations, we held a workshop on 23 and 24 May 2013, which was attended by the OPEN consortium (20 participants), and by an additional 14 key stakeholders from relevant key groups. During this workshop, each recommendation was discussed in a transparent and structured manner and subsequently rated as either ‘strong recommendation’ or ‘recommendation’ or ‘no recommendation (to be deleted)’.
To structure the discussion, to ensure consideration of all relevant aspects and to allow documentation of judgments, a decision table incorporating 5 rating criteria was used for rating each recommendation according to whether this particular recommendation was to be supported strongly by the workshop participants or not. A decision table is given in Appendix A.
During discussion of these draft recommendations, we decided to drop 23 recommendations and to add a further 2 recommendations. The 51 resulting final recommendations (4 general recommendations aimed at all or most key stakeholders in the knowledge translation process and 47 recommendations aimed at the specific target groups) on which a consensus was reached are given in here.

a. General recommendations
[1] We strongly recommend that all stakeholders in the knowledge generation and translation process raise awareness about dissemination bias and measures to reduce it.
[2] We strongly recommend that all stakeholders in the knowledge generation and translation process disseminate and facilitate implementation of targeted OPEN recommendations amongst key stakeholders of their respective group.
[3] We strongly recommend that all stakeholders in the knowledge generation and translation process promote trial registration and support initiatives that facilitate searches across multiple trial registries.
[4] We strongly recommend that all stakeholders in the knowledge generation and translation process support activities to systematically synthesize information from studies with rigorous methodology.

b. Targeted recommendations
1 Funding agencies
[1.1] We strongly recommend that funding agencies include a statement on dissemination bias and the requirement for the dissemination of research results in all calls for proposals.
[1.2] We strongly recommend that funding agencies include the requirement for grantees to provide a dissemination plan for funded projects in all calls for proposals.
[1.3] We strongly recommend that funding agencies include the requirement for grantees to explicitly declare that the results of funded research will be disseminated, regardless of the nature of findings, in all funding contracts.
[1.4] We recommend that funding agencies implement measures to ensure that the evaluation process of funded projects does not end with the project’s final report, but instead is followed up until all agreed data have been disseminated.
[1.5] We recommend that funding agencies consider providing incentives for researchers who disseminate their results, or, alternatively, withhold a part of the funding until a project’s results are adequately disseminated.
[1.6] We recommend that funding agencies create a publicly accessible database of all grants awarded and on how their results were disseminated in order to keep an accurate record of funded projects and dissemination outcomes.

2 Pharmaceutical and device industry
[2.1] We strongly recommend that pharmaceutical and devices companies make their policies concerning the dissemination of methods and results of clinical trials publicly accessible.
[2.2] We strongly recommend that pharmaceutical and devices companies register all clinical trials in a public registry before the recruitment of the first participant.
[2.3] We strongly recommend that pharmaceutical and devices companies make their study protocols & amendments (as submitted to ethical committee) available upon dissemination of results.
[2.4] We recommend that pharmaceutical and devices companies disseminate summary results of all trials conducted and provide access to their clinical study reports (CSR) (for clinical trials) upon request.

3 Research institutions
[3.1] We strongly recommend that research institutions provide guidance & training about the implications of and possible measures for avoiding dissemination bias.
[3.2] We strongly recommend that research institutions do not accept any funding that includes clauses that prevent the dissemination of data.
[3.3] We strongly recommend that research institutions mandate the dissemination of all clinical trial results.

4 Researchers I (Systematic Reviewers)
[4.1] We strongly recommend that researchers conducting systematic reviews (SRs), meta-analyses and network meta-analyses follow the best practices for performing SRs (especially those practices concerning the search for trials and the assessment of the impact of dissemination bias).
[4.2] We strongly recommend that all SR protocols and results of SRs informing clinical care are made publicly available.

5 Researchers II (Trialists)
[5.1] We strongly recommend that all trialists register every trial they conduct before the recruitment of the first participant.
[5.2] We strongly recommend that trialists disseminate the results from all clinical trials they conduct, e.g. through journal publications, results posting on registries, data sharing through data repositories.
- Summary results for all protocol-specified outcomes based on analyses of all participants (asap but 12 months after completion at the latest)
- Individual patient data available through data repositories, safeguard with regard to participant privacy/anonymity, ethical and scientific integrity.
[5.3] We strongly recommend that trialists make trial protocols publicly available (free of charge and in an easily accessible format) (ideally both within the register where the trial is registered and as appendix/supporting material with the journal publication).

6 Research ethics committees
[6.1] We strongly recommend that research ethics committees require the registration of all clinical trials before the recruitment of the first participant.
[6.2] We recommend that research ethics committees require that applicants commit to making all results publicly available
[6.3] We strongly recommend that research ethics committees encourage applicants to share anonymized individual patient level data on request.
[6.4] We strongly recommend that research ethics committees require that applicants provide annual reports describing the dissemination of their study results.

7 Trial registries
[7.1] We strongly recommend that trial registries enable the reporting of aggregate summary results.
[7.2] We recommend that trial registries enable and encourage the registration of observational human studies.
[7.3] We strongly recommend that trial registries outside English-speaking countries facilitate the registration process for non-English speaking trialists.
[7.4] We recommend the development of initiatives that enable non-English speakers to use the information contained in trial registries.
[7.5] We strongly recommend that trial registries enable and encourage the inclusion of links to publications and other permanent data sources (e.g. PubMed, other bibliographical databases, data repositories) in trial registry entries.
[7.6] We strongly recommend that all trial registries comply with the WHO International Standards for Clinical Trials Registries.

8 Journal editors / publishers
[8.1] We strongly recommend that journal editors and publishers remove all barriers to publishing negative or inconclusive studies and consider studies regardless of the direction of their findings or their sources of funding.
[8.2] We strongly recommend that all journals make trial registration a requirement for publication
[8.3] We strongly recommend that journal editors check all submitted manuscripts against trial registries or published protocols to detect selective reporting.
[8.4] We recommend that journal editors publish editorials and commentaries about the problem of dissemination bias and the benefits of trial registration.
[8.5] We strongly recommend that journals check for redundant publication of results by using text-matching software and asking peer reviewers about papers reporting the same findings.

9 Regulatory agencies
[9.1] We strongly recommend that the regulation of pharmaceutical products be extended to cover other therapeutic and diagnostic agents such as medical devices and biologicals.
[9.2] We strongly recommend that the responsible authorities (such as the EMA for drugs) mandate that all clinical trials in humans falling under their remit are registered in an EU database which is publicly accessible.
[9.3] We strongly recommend that the responsible authorities (such as EMA for drugs) mandate that upon a trial’s registration in an EU database, the full protocol approved by the research ethics committee, including the potential protocol amendments, is submitted and made publicly available as a searchable document.
[9.4] We strongly recommend that the responsible authorities (such as EMA for drugs) mandate that the full report including all results (e.g. Clinical Study Report) of a trial is made available in the same registry that the trial was registered in in a timely fashion (i.e. 1 year after study completion or inactivity) for all studies registered in the EU database.
[9.5] We strongly recommend that the responsible authorities (such as EMA for drugs) ensure that trial sponsors failing to comply with result submission requirements are sanctioned.

10 Benefit assessment institutions
[10.1] We strongly recommend that all benefit assessment institutions make their methods and processes of benefit assessment publicly available in order to achieve better transparency and understanding.
[10.2] We strongly recommend that benefit assessment institutions aim for a higher degree of collaboration between institutions to facilitate the detection of further (unpublished) data and to foster data sharing.
[10.3] We strongly recommend that benefit assessment institutions use the full evidence base on an intervention for their assessments.
[10.4] We strongly recommend that benefit assessment institutions specify their course of action if they find that the evidence base for an assessment is deemed incomplete (e.g. no adequate proof of benefit based on incomplete data set).
[10.5] We strongly recommend that benefit assessment institutions request from legislators the following items which will allow the consideration of all study results (disclosure of full protocols and full clinical study reports)
a. legal obligation for manufacturers to submit all requested evidence
b. public access to EMA databases
c. public access to protocols and full study reports

11 Legislators
[11.1] We strongly recommend that legislators make prospective registration of clinical trials in humans mandatory.
[11.2] We strongly recommend that legislators ensure that all data related to public and patient health are NOT commercially confidential/ proprietary information.
[11.3] We strongly recommend that legislators institute a legal obligation for manufacturers to submit all data and other required information for the formal decision-making process.
[11.4] We strongly recommend that legislators ensure that the raw data (anonymized individual patient data) are made publicly available for all clinical studies (registered in the EU database).

Summary
Our results depict the current views and practices of all key groups involved in knowledge generation and translation. Thereby, all our surveys used a similar methodology and were conducted at the same time; thus, our results offer a comprehensive picture of the current state of affairs. We are confident that the recommendations derived therefrom address the problem of dissemination bias in an exhaustive and up-to-date way.
Our results show that dissemination bias has been encountered by all key groups in the knowledge generation and translation process and that it is a very prevalent problem as shown by our systematic reviews on extent and impact of dissemination bias.
Our results also reflect the interdependency of the various groups in the dissemination process in tackling this problem. As an example for the interrelationships of the key groups, the recommendations for all key groups with regard to trial registration are depicted in the graph attached – trial registration needs to be enforced by all key groups to achieve a change in practice.

However, across all groups, there is a lack of awareness of the problems dissemination bias poses to science and society. Also, a tendency to shift responsibility to other key groups can be deduced by our survey findings. Thus, it is important to stress that common and concerted actions need to be taken.

Potential Impact:
The societal impact we hope to achieve is divided into two distinct parts: Firstly, we aim to raise awareness of the problem of dissemination bias and possible ways to tackle it, which constitutes a prerequisite for change; once awareness has been achieved, we hope for our recommendations to be considered and implemented by the respective groups in the knowledge translation and generation process, resulting in less dissemination bias.

1. Awareness of dissemination bias
Our survey results show that awareness of the problem of dissemination bias has not yet been sufficiently achieved; i.e. the disastrous consequences arising from it and practices by the various groups contributing to it are not well known among the key groups in a position to tackle this problem. Thus, the need for change is not yet perceived.
Therefore, we will disseminate as widely as possible the results of our surveys on opinions and views on this phenomenon as well as the current evidence on dissemination bias, and emphasize in each publication the negative consequences arising from the status quo.
We will also refer to our recommendations in our dissemination measures and so highlight that solutions are well possible and yet easily implemented, thereby hopefully motivating change.
Once awareness has been fostered, we are certain that a shift in thinking and a readiness for change will occur. This process will be facilitated by the fact that our results depict in a transparent fashion specifically which stakeholder needs to be responsible for doing what to reduce the problem of dissemination bias and thus offer a proactive vision of tackling this problem, instead of “shifting responsibility” to other parties as has been practice.
We are confident to have reached or to reach a sufficient level of awareness by means of
• Multiple publications in peer-reviewed journals (both in journals targeted at specific groups, e.g. addressing the research ethics committees, and in non-specific journals).
• Presentations at various meetings/conferences
• Further dissemination measures such as a project website, several newsletters and coverage in blogs
• Our surveys (which were sent out to a very large number and a broad range of participants)
• Attendance of our Workshop during which a lively discussion was fostered
For details on our dissemination plans, please see the next page.

2. Consideration and implementation of recommendations
The next step we hope for is consideration and implementation of our recommendations, leading to changes in practice, e.g. a policy change by RECs and trial registries.
Despite limited time, the complexity of the issue and differing opinions, we were able to discuss all of our recommendations with a broad range of stakeholders across all groups in the knowledge translation process. We are confident that by incorporating the views and possible objections of the various stakeholders, our recommendations stand a realistic chance of actually being implemented; also, acceptance of our recommendations among all key groups has been strengthened.

Our recommendations show that each key group can and should contribute to change - thus not “raising a finger” or even blaming single groups but rather promoting the understanding of this complex and interwoven system so that ideally, a concerted effort of all key groups will be made. Through our workshop, networking of the different key groups (which probably would ordinarily not have been in contact with each other) has been fostered and their views have been made plain; thus, a common effort all key groups has become more likely with regard to implementation.
Implementation of our recommendations will, among others, lead to more trials being registered, summary results posting in trial registries being made available, less papers being rejected due to the nature, direction or strength of the findings and funding agencies emphasizing the need for dissemination of results. Consequently, more data/results will be published in a journal or be made available to the public or the research community, resulting in a more comprehensive evidence base for medical decision making and, most importantly, in less patient harm and more patient benefit.
Less dissemination bias would also imply less waste of resources, and a more strategic research funding in that no previously conducted but unpublished studies would be replicated. It is also very likely that reimbursement decisions by national HTA institutes will be affected due to less biased systematic reviews so that societies will no longer pay for ineffective treatments.
Furthermore, less failure to fully publish the results of clinical research reestablishes ethics in medical research – patients who have consented to participate in a trial, because they are led to believe that such research is necessary for the advancement of medical science, should be respected for their personal contribution by disseminating the results of trials. Also, people will be more willing to volunteer in clinical trials and public trust in clinical research will be reinstated.
Some of our recommendations can be implemented very easily – if they are taken up by the respective groups, changes in practice can be achieved within a very short time frame.

3. Further impact
However, improvements can be achieved best if implementation of our recommendations is enforced with appropriate sanctioning. Therefore, we want to encourage legislators to consider our project’s results and our recommendations.
The final recommendations from the OPEN project coincide with a number of important decisions by the EMA and the European Parliament: e.g. EMA is expected to publish its final policy on publication and access to clinical-trial data by the end of 2013 [11], and the European Parliament is awaited to have a first reading on its Clinical Trials Directive/Regulation [12].
We hope that our project can contribute to these decisions by highlighting what is important to consider when making these legislations and by once again emphasizing the need for unbiased and full dissemination of results of clinical trials.

4. Main dissemination activities
We tried to maximize dissemination of OPEN’s results and the final recommendations listed in this report by pursuing the following activities:
4.1 OPEN website
A website about the OPEN project was established on 01/11/2011 which was updated regularly throughout the course of the project. Its content includes the scientific background & rationale of the project, the project partners and contact details plus a list of related publications and presentations as well as all newsletters. Also, a mailing list for interested visitors was established.
The website will be updated with a list and explanation of all final and consented recommendations.
User numbers mirror the interest in the OPEN project: Since July 2012 (when we started tracking site usage), 4.489 people visited OPEN’s project website.
This website will be maintained for several years to come to allow for interactive information on the project and its results, including the publications to come.

4.2 Journal publications
As of now, the OPEN consortium has published 10 manuscripts in 5 journals on the methods and results of OPEN, including an editorial and 3 publications in high-impact journals (BMJ, PLoS One) [1-10]. One publication (in preparation) focuses specifically on the process of deriving the recommendations and lists all final and consented recommendations along with a rationale behind them. All publications were in open-access journals; all protocols of the systematic reviews conducted were published to enhance transparency of research.
Further publications, emphasizing on the survey results, are currently being prepared and/or submitted for publication.

4.3 Further dissemination efforts
The OPEN project was presented at several scientific meetings (including the Cochrane Colloquium 2013), and further presentations are planned.
2 newsletters were sent to interested persons and made available for download on OPEN’s website – a third newsletter is currently in preparation.
Furthermore, the OPEN project was mentioned in a newsletter of the European Menopause and Andropause Society which is read by more than 30.000 people and was covered in a piece in a national (Croatian) newspaper.
We are currently planning further dissemination activities including a press release on CORDIS, and further coverage in non-specialist media.
In addition, our consortium has excellent contacts with key opinion leaders of all relevant key groups in the knowledge translation process. Through these contacts, we will spread knowledge of OPEN’s results to the respective groups.

List of Websites:
http://www.open-project.eu/

Final Report Summary - OPEN (Overcome failure to Publish nEgative fiNdings)

Related documents

Share this page

Download