Final Report Summary - COMET (The COMET (Core Outcome Measures in Effectiveness Trials) Initiative)
Executive Summary:
The COMET (Core Outcome Measures in Effectiveness Trials) Initiative was launched in January 2010, amidst a growing recognition that insufficient attention had been paid to the selection of outcomes to measure in clinical trials and clinical audit. This had led to waste and a failure to capitalise on the potential power of research to improve health care and health. Core outcome sets (COS) can help overcome this. They should be developed with input from all relevant stakeholders, including health service users, to make them meaningful for all decision makers. COS should be measured and reported in all trials in a specific health condition, along with other outcomes deemed important by those involved in the research. Greater harmonization should lead to the greater influence of research findings on practice and, therefore, speed up the translation of research into health gains. This FP7 Coordinating and Support Action supported work from 2012 to 2014 to further the strategic development of COMET, bringing together COS developers, researchers, public and patient involvement organisations, health service users, policy makers, funders, regulators, journal editors and others, to develop a roadmap for the selection and measurement of outcomes, which could not be achieved working in isolation. This helped COMET: (i) to raise awareness of current problems with outcomes in clinical trials; (ii) to encourage COS development and uptake; (iii) to provide resources to facilitate this; and (iv) to encourage evidence-based COS development.
A key element in COMET’s communication strategy is our website, housing a database of all published and ongoing COS projects, as well as a variety of other materials to support the development of COS. Over the last two years, the database has been enhanced considerably, drawing on our systematic review which identified nearly 200 COS studies, covering many areas of health, and highlighting important gaps where future COS are needed. COMET has an important role in providing guidance and support for these COS and we undertook various activities that will help with this. We developed a repository of relevant work to date, and are collating and developing evidence-based methodological and reporting guidelines. Our direct involvement with the development of three COS confirmed the value of personal advice from experienced COS developers and our network meetings brought together COS developers, allowing the discussion of different methodologies and the exchange of experiences. The international COMET meetings also provided opportunities for networking activities.
We have engaged with and supported organisations able to influence the development and uptake of COS in trials, systematic reviews, and guidelines; and will continue to do so. This raises awareness among people developing COS, public and patient involvement organisations, researchers, health service users, practitioners, policy makers, funders, regulators, and journal editors. For example, we are advocating for, and providing advice on, the involvement of health service users in COS studies, and 90% of the 49 ongoing projects known to COMET include patients, carers, or patient support group representatives as participants, compared to only 16% of those published up to June 2013.
With regard to systematic reviews and guidelines, our survey of 1200 Cochrane Reviews published for the first time in 2007, 2011 and 2013 found improvements in the quality of reporting of outcomes and the use of Summary of Findings tables in Cochrane Reviews: from 31% in 2011 to 57% in 2013. We are working with the Cochrane Informatics and Knowledge Management Department on the implementation of a COMET taxonomy (derived from this survey and our systematic review) to code the outcomes in Cochrane Reviews and their included studies. This taxonomy will be considered for the software for producing Cochrane Reviews (RevMan), including Summary of Findings tables, and would allow linking to the COS in the COMET database.
We predict that the number of new COS published per year will double from about 20 to about 40 by 2018, due in part to advocacy by COMET. We have developed a model of workforce support for COS development and will seek funding to implement this. It includes ‘self-help’ resources, personal advice from experienced COS developers, and network meetings that will bring together COS developers from different fields.
Project Context and Objectives:
(The scientific officer for this project Cornelius Schmaltz has agreed that because this project consisted of a single period, the final publishable summary report is identical to the publishable summary of your periodic report)
Insufficient attention has been paid to the selection of outcomes to measure in clinical trials and clinical audit. This has led to waste and a failure to capitalise on the potential power of research to improve health care and health. Outcomes need to be relevant to health service users and others making choices about care if research is to have an influence. Standardisation is necessary to contrast or combine data from different studies in evidence synthesis and to make comparisons between data sets. Consensus is also needed on how outcomes should be defined and measured, and guidance is needed on how to select the best instruments. Different instruments are often used for the same outcome, with great variability in how outcomes are measured and reported. Incomparable data from different instruments restricts evidence synthesis and hampers decision makers who wish to use research. Selective reporting is a further problem. Selecting a subset of outcomes to publish because of the results obtained is a significant problem, leading to bias in the available evidence. All these issues could be addressed through agreed standardised sets of outcomes, known as ‘core outcome sets’ (COS). These should be developed with input from all relevant stakeholders, including health service users, to make them meaningful for all decision makers. COS should be measured and reported in all trials in a specific health condition, along with other outcomes deemed important by those involved in the research.
The development of COS is an important emerging area of health research which needs to be accelerated and undertaken in ways that maximise efficiency. To improve the quality of research-based knowledge, COS for particular health conditions need to be implemented across Europe and the rest of the world. COS will shape the future of European and international clinical trials practice, increasing the efficiency and value of the research by reducing the inefficiencies of trials of similar topics measuring outcomes in different ways. This should lead to more efficient use of national health research budgets and decrease the costs of commercial research by making the selection and development of outcome measures easier. COS will help avoid the waste of research that is ignored or needs to be repeated because the chosen outcomes were not sufficiently relevant. Greater harmonization should lead to the greater influence of the findings of research on practice and, therefore, speed up the translation of research into health gains.
The COMET Initiative was launched in January 2010, with the following aims:
(i) To raise awareness of current problems with outcomes in clinical trials;
(ii) To encourage COS development and uptake;
(iii) To provide resources to facilitate this;
(iv) To encourage evidence-based COS development.
This FP7 Coordinating and Support Action sought to further the strategic development of the COMET Initiative. It brought together groups working in complementary methodological areas and other international partners, to develop a roadmap for the selection and measurement of outcomes in clinical trials and systematic reviews, which could not be achieved working in isolation. Tasks included the development of a publically available resource and a searchable database of studies developing COS; provision of advice to researchers developing COS; delivery of workshops on COS and evidence-based instrument selection at key national and international meetings; preparation of guidance on developing and using COS, and on how to select instruments to measure COS outcomes; and the forging or strengthening of partnerships with COS developers and users.
Project Results:
(The scientific officer for this project Cornelius Schmaltz has agreed that because this project consisted of a single period, the final publishable summary report is identical to the publishable summary of your periodic report)
Our systematic review identified projects to determine which outcomes or domains to measure in clinical trials in a specific condition and studies of the methods used. We identified 198 eligible studies, covering various areas of health, and highlighting important gaps. The review shows that a variety of methods have been used to develop COS and that the most common stakeholders involved were clinical experts and non-clinical research experts.
The credibility of a COS depends on the use of sound methods for its development and transparent reporting of these methods. Its implementation in trials will depend on effective dissemination and acceptance by the research community including researchers, patients and the public, and funders. Our review showed the need for methodological guidance, including how to engage stakeholder groups, particularly the public. We identified several issues to be considered in COS development and recommend: a protocol be developed before the study, and made publically available; the COS should be registered in a public repository, such as the COMET database; methodological research should be done to improve the methods, including nesting research within COS development.
In addition to deciding what to measure, consideration is needed to how COS outcomes should be defined and measured. Our Delphi consensus study (95 experts) identified the steps needed to find and select outcome measurement instruments for a COS. We wrote a draft guideline to provide guidance on this for COS developers, including the essential information that should be gathered for each measurement instrument identified (eg. characteristics of the instrument and information on validity and reliability); how to evaluate the quality of the identified instruments and select the most appropriate ones. We developed and populated an online database of 572 systematic reviews of outcome measurement instruments (www.database.cosmin.nl).
We surveyed new Cochrane Reviews from 2007 (387 reviews), 2011 (401) and 2013 (439), extracting information on outcomes specified in the methods section and outcomes reported in the results section of each review to explore the common use of outcomes and COS. There were more than 6000 outcomes in the initial survey (2007 and 2011 combined) and 3142 in 2013. We found that the quality of reporting of outcomes is improving over time, with 61%, 65% and 68% of specified outcomes being reported in each year. The use of Summary of Findings tables in Cochrane Reviews is also increasing, from 31% in 2011 to 57% in 2013. This and other work helped us strengthen our engagement with Cochrane and GRADE, to support joint efforts to improve the use of COS in reviews and guidelines.
We undertook various activities to evaluate a model of workforce support for COS development, and to assess the resource requirements to make this sustainable for COMET. In relation to ‘self-help’ resources, we developed a repository of relevant work to date, and are collating and developing methodological and reporting guidelines. We anticipate the development of a COMET Handbook, to provide evidence-based recommendations for COS methodology. Our direct involvement with the development of three COS confirmed that personal advice from experienced COS developers is very helpful, especially in the early stages. A key challenge for COMET is to continue to increase the number of COS developers willing and able to advise others. For personal support to COS developers to be upscaled, more experts are needed and funding is needed to train them. Our four network meetings (Amsterdam, Bristol, Manchester and Washington) brought together COS developers from different fields, allowing the discussion of different methodologies for COS development, and the exchange of experiences. The international COMET meetings also provided opportunities to arrange networking activities.
Potential Impact:
(The scientific officer for this project Cornelius Schmaltz has agreed that because this project consisted of a single period, the final publishable summary report is identical to the publishable summary of your periodic report)
We predict that the number of COS published per year will double from about 20 now to about 40 by 2018, in large part because of advocacy by COMET. We wish to continue to provide support for these COS developers, but funding is needed to develop and maintain the self-help resources, to train COS advisors and to organize COMET conferences to support networking. Our work to date will also continue to have an impact as described here.
A key output of COMET is our website, providing a database of all published and ongoing COS projects, resources to assist COS developers, plain language materials, information on activities and events, and links to other initiatives. Use of the website is increasing, with more than 16,500 visits in 2014 (36% more than 2013) and 9780 new visitors (43% increase). Visitors are from many countries: rising from 66 countries in 2011 to 124 in 2014. In 2014, 49% of visitors were in the UK, 16% were in the USA and Canada, and 35% were in other countries. By the end of 2014, the COMET database included a unique collection of 567 studies, a growth of 261 since 2013. A batch of COS published up to the end of 2014 is being added, keeping the resource up to date. 6588 database searches have been run since its inception (2383 of these were in 2014).
We will continue to engage with and support organisations able to influence the development and uptake of COS in trials, systematic reviews, and guidelines. We are raising awareness among people developing COS, public and patient involvement organisations, researchers, health service users, practitioners, policy makers, funders, regulators, and journal editors. We have met with several groups with an interest in outcomes, e.g. EMA, FDA, EUPATI, ECRIN, clinicaltrials.gov CDISC/CFAST, EURODIS, and EFPIA. Our review of COS was sent to 245 funders in 31 countries and further targeted communication is planned. We will continue to advocate for, and provide advice on, the involvement of health service users in COS studies, and 90% of the 49 ongoing projects known to COMET include patients, carers, or patient support group representatives as participants, compared to only 16% of those published up to June 2013. Looking beyond the use of COS in trials, systematic reviews and guidelines, COMET is building links with groups considering the implementation and evaluation of COS in routine care and value-based commissioning.
Our knowledge exchange activities included several workshops for COS developers and users at national and international events, including the COMET III (2013) and COMET IV (2014) meetings. The excellent feedback suggests that this learning will be put to good use in coming years.
Our ongoing engagement will help the development and uptake of COS, increasing the efficiency and value of research by reducing the waste that arises when trials in the same clinical area measure outcomes in different ways. Greater harmonization should lead to greater influence of the findings of research on practice and speed up the translation of research into health gains. For example, we have shown the marked increase in the measurement of the full set of core outcomes in pharmacologic trials after publication of a COS for rheumatoid arthritis: from around 40% just before publication to 70% 6 years later. Most trialists who had not reported the full COS were unaware of it when designing their trial, but all said they would consider implementing it in a new trial.
Our discussions with the Informatics and Knowledge Management Department of the Cochrane Collaboration has led to the use of a COMET taxonomy (derived from our systematic review and survey of Cochrane Reviews) to code the outcomes in Cochrane Reviews and their included studies. This taxonomy will be considered for the next generation of software for producing Cochrane Reviews (RevMan), including Summary of Findings tables. This would support real-time linking to the COS in the COMET database.
List of Websites:
http://www.comet-initiative.org/
The COMET (Core Outcome Measures in Effectiveness Trials) Initiative was launched in January 2010, amidst a growing recognition that insufficient attention had been paid to the selection of outcomes to measure in clinical trials and clinical audit. This had led to waste and a failure to capitalise on the potential power of research to improve health care and health. Core outcome sets (COS) can help overcome this. They should be developed with input from all relevant stakeholders, including health service users, to make them meaningful for all decision makers. COS should be measured and reported in all trials in a specific health condition, along with other outcomes deemed important by those involved in the research. Greater harmonization should lead to the greater influence of research findings on practice and, therefore, speed up the translation of research into health gains. This FP7 Coordinating and Support Action supported work from 2012 to 2014 to further the strategic development of COMET, bringing together COS developers, researchers, public and patient involvement organisations, health service users, policy makers, funders, regulators, journal editors and others, to develop a roadmap for the selection and measurement of outcomes, which could not be achieved working in isolation. This helped COMET: (i) to raise awareness of current problems with outcomes in clinical trials; (ii) to encourage COS development and uptake; (iii) to provide resources to facilitate this; and (iv) to encourage evidence-based COS development.
A key element in COMET’s communication strategy is our website, housing a database of all published and ongoing COS projects, as well as a variety of other materials to support the development of COS. Over the last two years, the database has been enhanced considerably, drawing on our systematic review which identified nearly 200 COS studies, covering many areas of health, and highlighting important gaps where future COS are needed. COMET has an important role in providing guidance and support for these COS and we undertook various activities that will help with this. We developed a repository of relevant work to date, and are collating and developing evidence-based methodological and reporting guidelines. Our direct involvement with the development of three COS confirmed the value of personal advice from experienced COS developers and our network meetings brought together COS developers, allowing the discussion of different methodologies and the exchange of experiences. The international COMET meetings also provided opportunities for networking activities.
We have engaged with and supported organisations able to influence the development and uptake of COS in trials, systematic reviews, and guidelines; and will continue to do so. This raises awareness among people developing COS, public and patient involvement organisations, researchers, health service users, practitioners, policy makers, funders, regulators, and journal editors. For example, we are advocating for, and providing advice on, the involvement of health service users in COS studies, and 90% of the 49 ongoing projects known to COMET include patients, carers, or patient support group representatives as participants, compared to only 16% of those published up to June 2013.
With regard to systematic reviews and guidelines, our survey of 1200 Cochrane Reviews published for the first time in 2007, 2011 and 2013 found improvements in the quality of reporting of outcomes and the use of Summary of Findings tables in Cochrane Reviews: from 31% in 2011 to 57% in 2013. We are working with the Cochrane Informatics and Knowledge Management Department on the implementation of a COMET taxonomy (derived from this survey and our systematic review) to code the outcomes in Cochrane Reviews and their included studies. This taxonomy will be considered for the software for producing Cochrane Reviews (RevMan), including Summary of Findings tables, and would allow linking to the COS in the COMET database.
We predict that the number of new COS published per year will double from about 20 to about 40 by 2018, due in part to advocacy by COMET. We have developed a model of workforce support for COS development and will seek funding to implement this. It includes ‘self-help’ resources, personal advice from experienced COS developers, and network meetings that will bring together COS developers from different fields.
Project Context and Objectives:
(The scientific officer for this project Cornelius Schmaltz has agreed that because this project consisted of a single period, the final publishable summary report is identical to the publishable summary of your periodic report)
Insufficient attention has been paid to the selection of outcomes to measure in clinical trials and clinical audit. This has led to waste and a failure to capitalise on the potential power of research to improve health care and health. Outcomes need to be relevant to health service users and others making choices about care if research is to have an influence. Standardisation is necessary to contrast or combine data from different studies in evidence synthesis and to make comparisons between data sets. Consensus is also needed on how outcomes should be defined and measured, and guidance is needed on how to select the best instruments. Different instruments are often used for the same outcome, with great variability in how outcomes are measured and reported. Incomparable data from different instruments restricts evidence synthesis and hampers decision makers who wish to use research. Selective reporting is a further problem. Selecting a subset of outcomes to publish because of the results obtained is a significant problem, leading to bias in the available evidence. All these issues could be addressed through agreed standardised sets of outcomes, known as ‘core outcome sets’ (COS). These should be developed with input from all relevant stakeholders, including health service users, to make them meaningful for all decision makers. COS should be measured and reported in all trials in a specific health condition, along with other outcomes deemed important by those involved in the research.
The development of COS is an important emerging area of health research which needs to be accelerated and undertaken in ways that maximise efficiency. To improve the quality of research-based knowledge, COS for particular health conditions need to be implemented across Europe and the rest of the world. COS will shape the future of European and international clinical trials practice, increasing the efficiency and value of the research by reducing the inefficiencies of trials of similar topics measuring outcomes in different ways. This should lead to more efficient use of national health research budgets and decrease the costs of commercial research by making the selection and development of outcome measures easier. COS will help avoid the waste of research that is ignored or needs to be repeated because the chosen outcomes were not sufficiently relevant. Greater harmonization should lead to the greater influence of the findings of research on practice and, therefore, speed up the translation of research into health gains.
The COMET Initiative was launched in January 2010, with the following aims:
(i) To raise awareness of current problems with outcomes in clinical trials;
(ii) To encourage COS development and uptake;
(iii) To provide resources to facilitate this;
(iv) To encourage evidence-based COS development.
This FP7 Coordinating and Support Action sought to further the strategic development of the COMET Initiative. It brought together groups working in complementary methodological areas and other international partners, to develop a roadmap for the selection and measurement of outcomes in clinical trials and systematic reviews, which could not be achieved working in isolation. Tasks included the development of a publically available resource and a searchable database of studies developing COS; provision of advice to researchers developing COS; delivery of workshops on COS and evidence-based instrument selection at key national and international meetings; preparation of guidance on developing and using COS, and on how to select instruments to measure COS outcomes; and the forging or strengthening of partnerships with COS developers and users.
Project Results:
(The scientific officer for this project Cornelius Schmaltz has agreed that because this project consisted of a single period, the final publishable summary report is identical to the publishable summary of your periodic report)
Our systematic review identified projects to determine which outcomes or domains to measure in clinical trials in a specific condition and studies of the methods used. We identified 198 eligible studies, covering various areas of health, and highlighting important gaps. The review shows that a variety of methods have been used to develop COS and that the most common stakeholders involved were clinical experts and non-clinical research experts.
The credibility of a COS depends on the use of sound methods for its development and transparent reporting of these methods. Its implementation in trials will depend on effective dissemination and acceptance by the research community including researchers, patients and the public, and funders. Our review showed the need for methodological guidance, including how to engage stakeholder groups, particularly the public. We identified several issues to be considered in COS development and recommend: a protocol be developed before the study, and made publically available; the COS should be registered in a public repository, such as the COMET database; methodological research should be done to improve the methods, including nesting research within COS development.
In addition to deciding what to measure, consideration is needed to how COS outcomes should be defined and measured. Our Delphi consensus study (95 experts) identified the steps needed to find and select outcome measurement instruments for a COS. We wrote a draft guideline to provide guidance on this for COS developers, including the essential information that should be gathered for each measurement instrument identified (eg. characteristics of the instrument and information on validity and reliability); how to evaluate the quality of the identified instruments and select the most appropriate ones. We developed and populated an online database of 572 systematic reviews of outcome measurement instruments (www.database.cosmin.nl).
We surveyed new Cochrane Reviews from 2007 (387 reviews), 2011 (401) and 2013 (439), extracting information on outcomes specified in the methods section and outcomes reported in the results section of each review to explore the common use of outcomes and COS. There were more than 6000 outcomes in the initial survey (2007 and 2011 combined) and 3142 in 2013. We found that the quality of reporting of outcomes is improving over time, with 61%, 65% and 68% of specified outcomes being reported in each year. The use of Summary of Findings tables in Cochrane Reviews is also increasing, from 31% in 2011 to 57% in 2013. This and other work helped us strengthen our engagement with Cochrane and GRADE, to support joint efforts to improve the use of COS in reviews and guidelines.
We undertook various activities to evaluate a model of workforce support for COS development, and to assess the resource requirements to make this sustainable for COMET. In relation to ‘self-help’ resources, we developed a repository of relevant work to date, and are collating and developing methodological and reporting guidelines. We anticipate the development of a COMET Handbook, to provide evidence-based recommendations for COS methodology. Our direct involvement with the development of three COS confirmed that personal advice from experienced COS developers is very helpful, especially in the early stages. A key challenge for COMET is to continue to increase the number of COS developers willing and able to advise others. For personal support to COS developers to be upscaled, more experts are needed and funding is needed to train them. Our four network meetings (Amsterdam, Bristol, Manchester and Washington) brought together COS developers from different fields, allowing the discussion of different methodologies for COS development, and the exchange of experiences. The international COMET meetings also provided opportunities to arrange networking activities.
Potential Impact:
(The scientific officer for this project Cornelius Schmaltz has agreed that because this project consisted of a single period, the final publishable summary report is identical to the publishable summary of your periodic report)
We predict that the number of COS published per year will double from about 20 now to about 40 by 2018, in large part because of advocacy by COMET. We wish to continue to provide support for these COS developers, but funding is needed to develop and maintain the self-help resources, to train COS advisors and to organize COMET conferences to support networking. Our work to date will also continue to have an impact as described here.
A key output of COMET is our website, providing a database of all published and ongoing COS projects, resources to assist COS developers, plain language materials, information on activities and events, and links to other initiatives. Use of the website is increasing, with more than 16,500 visits in 2014 (36% more than 2013) and 9780 new visitors (43% increase). Visitors are from many countries: rising from 66 countries in 2011 to 124 in 2014. In 2014, 49% of visitors were in the UK, 16% were in the USA and Canada, and 35% were in other countries. By the end of 2014, the COMET database included a unique collection of 567 studies, a growth of 261 since 2013. A batch of COS published up to the end of 2014 is being added, keeping the resource up to date. 6588 database searches have been run since its inception (2383 of these were in 2014).
We will continue to engage with and support organisations able to influence the development and uptake of COS in trials, systematic reviews, and guidelines. We are raising awareness among people developing COS, public and patient involvement organisations, researchers, health service users, practitioners, policy makers, funders, regulators, and journal editors. We have met with several groups with an interest in outcomes, e.g. EMA, FDA, EUPATI, ECRIN, clinicaltrials.gov CDISC/CFAST, EURODIS, and EFPIA. Our review of COS was sent to 245 funders in 31 countries and further targeted communication is planned. We will continue to advocate for, and provide advice on, the involvement of health service users in COS studies, and 90% of the 49 ongoing projects known to COMET include patients, carers, or patient support group representatives as participants, compared to only 16% of those published up to June 2013. Looking beyond the use of COS in trials, systematic reviews and guidelines, COMET is building links with groups considering the implementation and evaluation of COS in routine care and value-based commissioning.
Our knowledge exchange activities included several workshops for COS developers and users at national and international events, including the COMET III (2013) and COMET IV (2014) meetings. The excellent feedback suggests that this learning will be put to good use in coming years.
Our ongoing engagement will help the development and uptake of COS, increasing the efficiency and value of research by reducing the waste that arises when trials in the same clinical area measure outcomes in different ways. Greater harmonization should lead to greater influence of the findings of research on practice and speed up the translation of research into health gains. For example, we have shown the marked increase in the measurement of the full set of core outcomes in pharmacologic trials after publication of a COS for rheumatoid arthritis: from around 40% just before publication to 70% 6 years later. Most trialists who had not reported the full COS were unaware of it when designing their trial, but all said they would consider implementing it in a new trial.
Our discussions with the Informatics and Knowledge Management Department of the Cochrane Collaboration has led to the use of a COMET taxonomy (derived from our systematic review and survey of Cochrane Reviews) to code the outcomes in Cochrane Reviews and their included studies. This taxonomy will be considered for the next generation of software for producing Cochrane Reviews (RevMan), including Summary of Findings tables. This would support real-time linking to the COS in the COMET database.
List of Websites:
http://www.comet-initiative.org/
