Periodic Reporting for period 2 - RCT4MANU (Test an innovative support scheme for manufacturing SMEs and accelerate the use of RCTs in innovation agencies)
Okres sprawozdawczy: 2020-12-01 do 2022-08-31
The UK manufacturing sector is facing significant issues with productivity compared to similar sized economies in Europe. The arrival of ‘Industry 4.0’ (the automation of manufacturing practices using smart technologies) is seen as an opportunity to revolutionise manufacturing and take steps to improve productivity, it is also an opportunity for businesses to review and adopt more environmentally sustainable practices to support the movement to Net Zero.
To address this, since 2017 UK-based innovation agency Knowledge Transfer Network (KTN) have developed an advisor-led tool, 4Manufacturing®, to use as a framework to support businesses in identifying strategies to improve productivity and sustainability, predominantly driven by exploring options to adopt industrial digital technologies to address the key challenges faced by businesses and progress towards Industry 4.0.
The RCT4MANU study has been developed to robustly evaluate the effectiveness of the support on manufacturing SMEs and their subsequent adoption of technologies and the impact on their business as a result.
Further to this, there is a strong driver from policy makers to use an evidence-based model when it comes to policy decision making. As such, there is a strong focus within UK research & development funding agencies to ensure robust methodologies are used when evaluating policies and funding programmes. Another aim of the project is to develop learning within partner organisations on the design, execution and evaluation of randomised control trials and how these may be better utilised as an evaluation methodology within innovation agencies going forward.
To address this, since 2017 UK-based innovation agency Knowledge Transfer Network (KTN) have developed an advisor-led tool, 4Manufacturing®, to use as a framework to support businesses in identifying strategies to improve productivity and sustainability, predominantly driven by exploring options to adopt industrial digital technologies to address the key challenges faced by businesses and progress towards Industry 4.0.
The RCT4MANU study has been developed to robustly evaluate the effectiveness of the support on manufacturing SMEs and their subsequent adoption of technologies and the impact on their business as a result.
Further to this, there is a strong driver from policy makers to use an evidence-based model when it comes to policy decision making. As such, there is a strong focus within UK research & development funding agencies to ensure robust methodologies are used when evaluating policies and funding programmes. Another aim of the project is to develop learning within partner organisations on the design, execution and evaluation of randomised control trials and how these may be better utilised as an evaluation methodology within innovation agencies going forward.
The trial’s primary hypothesis was that firms receiving the 4Manufacturing® support would adopt 20% more technologies on average than the control group (this figure includes committed-to/approved adoptions within the 12 months following the end of the trial).
The project recruited 91 businesses to participate in the trial, 51 of which were offered 4Manufacturing® support, the remaining 40 formed the control group, who did not receive support during the trial, but were offered it after 12 months as an incentive to participate. The biggest challenge the project faced was recruitment; it had an initial target of recruiting over 300 businesses, significantly higher than was achieved. As the trial commenced at the end of 2020, low recruitment levels were affected by macro-economic challenges resulting from the Covid-19 pandemic and the UK’s exit from the European Union, however there may be other factors relating to the trial’s recruitment strategy that could potentially have improved uptake.
The low participant numbers made the likelihood of observing a powered, statistically significant result much less likely, and unfortunately this was the case when it came to analysing outcome data. The evidence is therefore inconclusive at this stage as to the effect of the tool on technology adoption.
As well as an impact evaluation, the team delivered a process evaluation of the support to better understand its perceived effectiveness from participant feedback. The evaluation provided useful insight and identified improvements to the tool and wrap around advisory support going forward, which are considerations for a potential scale-up of the 4Manufacturing® scheme in future.
The project recruited 91 businesses to participate in the trial, 51 of which were offered 4Manufacturing® support, the remaining 40 formed the control group, who did not receive support during the trial, but were offered it after 12 months as an incentive to participate. The biggest challenge the project faced was recruitment; it had an initial target of recruiting over 300 businesses, significantly higher than was achieved. As the trial commenced at the end of 2020, low recruitment levels were affected by macro-economic challenges resulting from the Covid-19 pandemic and the UK’s exit from the European Union, however there may be other factors relating to the trial’s recruitment strategy that could potentially have improved uptake.
The low participant numbers made the likelihood of observing a powered, statistically significant result much less likely, and unfortunately this was the case when it came to analysing outcome data. The evidence is therefore inconclusive at this stage as to the effect of the tool on technology adoption.
As well as an impact evaluation, the team delivered a process evaluation of the support to better understand its perceived effectiveness from participant feedback. The evaluation provided useful insight and identified improvements to the tool and wrap around advisory support going forward, which are considerations for a potential scale-up of the 4Manufacturing® scheme in future.
Policy implications from RCT4MANU.
Another aim of the project was to assess how the RCT methodology could be used in the evaluation of innovation support programmes and its implications for future policy decision making. The effort required in designing, running and evaluating an RCT is significant and requires specialist skillsets in evaluation and statistical testing. For many innovation agencies this is not something that is readily available and so resourcing should be a key consideration.
Another key learning point is that an RCT evaluation, while robust if executed successfully, is not necessarily the most appropriate choice of methodology. They are complex and time consuming to set up, and without appropriate expertise and resource are very difficult to deliver.
An RCT should be used to confirm effectiveness, rather than to explore it. Robust evidence and assumptions are required to develop hypotheses, and without these there is a risk of misidentifying, or overestimating the outputs and outcomes and therefore measuring the wrong thing. Recruitment and retention are the biggest risk areas, failure in this area can compromise the validity of the study.
Care should be taken to ensure that the RCT methodology does not adversely affect the way in which the intervention is delivered. In the case of RCT4MANU the RCT framework altered both the way in which SMEs were recruited (email campaign rather than by word of mouth) and when the support was delivered (all workshops delivered at the same time rather than as and when business required support). There was a delay from businesses signing up to the sessions being arranged and delivered which resulted in many participants dropping out because they had lost interest or moved on before sessions could be arranged.
There are ways to bring elements of RCT design to other methodologies to develop robust evaluations without running a full RCT. These include careful consideration of research questions, outputs and outcomes, upfront definition of metrics and data collection frameworks and communication plans. Establishing these practices early in the intervention lifecycle will increase the robustness of impact evidence, and can provide the necessary frameworks to conduct an RCT at a later point.
Another aim of the project was to assess how the RCT methodology could be used in the evaluation of innovation support programmes and its implications for future policy decision making. The effort required in designing, running and evaluating an RCT is significant and requires specialist skillsets in evaluation and statistical testing. For many innovation agencies this is not something that is readily available and so resourcing should be a key consideration.
Another key learning point is that an RCT evaluation, while robust if executed successfully, is not necessarily the most appropriate choice of methodology. They are complex and time consuming to set up, and without appropriate expertise and resource are very difficult to deliver.
An RCT should be used to confirm effectiveness, rather than to explore it. Robust evidence and assumptions are required to develop hypotheses, and without these there is a risk of misidentifying, or overestimating the outputs and outcomes and therefore measuring the wrong thing. Recruitment and retention are the biggest risk areas, failure in this area can compromise the validity of the study.
Care should be taken to ensure that the RCT methodology does not adversely affect the way in which the intervention is delivered. In the case of RCT4MANU the RCT framework altered both the way in which SMEs were recruited (email campaign rather than by word of mouth) and when the support was delivered (all workshops delivered at the same time rather than as and when business required support). There was a delay from businesses signing up to the sessions being arranged and delivered which resulted in many participants dropping out because they had lost interest or moved on before sessions could be arranged.
There are ways to bring elements of RCT design to other methodologies to develop robust evaluations without running a full RCT. These include careful consideration of research questions, outputs and outcomes, upfront definition of metrics and data collection frameworks and communication plans. Establishing these practices early in the intervention lifecycle will increase the robustness of impact evidence, and can provide the necessary frameworks to conduct an RCT at a later point.