Final Report Summary - ENGAGE (Equipping the Next Generation for Active Engagement in Science)
Executive Summary:
Shifting school science towards RRI
ENGAGE aimed to give the next generation of students the knowledge, skills and attitudes to deal with socio-scientific issues in their lives, and develop informed opinions on emerging science and technology. ENGAGE’s goal is to embed Responsible Research and Innovation (RRI) within the science curriculum and change the way science is taught. Through a set of innovative strategies, we have attempted to shift the emphasis from transmitting a body of scientific knowledge towards applying science to issues that matter to students. At the heart of ENGAGE is using authentic activities to simulate how citizens conduct inquiries. ENGAGE operates on a major scale. More than 15,000 teachers have signed up across 11 partner countries: UK. Greece, Germany, France, Romania, Israel, Spain, Norway, Switzerland, Lithuania, and Cyprus.
State of the art innovation
Teachers need make major changes in practice to use socio-scientific issues in the classroom. Therefore ENGAGE chose a thought-out, multi-pronged approach and created a series of positive conditions for change. At the heart of the project were high-quality lessons that would encourage RRI-based teaching through effective engagement. We also constructed an escalated pathway of progression in teaching skills load to foster teacher change- In the ENGAGE model, once teachers gain confidence with the intuitive ADOPT teaching approaches, a proportion progress to using more advanced ADAPT materials in the next stage. This second stage involves a deeper interest and commitment to the ENGAGE philosophy, because the materials demand more curriculum time to explicitly teach RRI skills. The third step, called TRANSFORM, is our experiment in project-based learning and an opportunity for students to develop their transferrable skills. It was designed for a small proportion of teachers who wanted to make RRI and socio-scientific issues a major focus of their curriculum enhancement. We provided support to help them plan an issue-based project where students could engage in more autonomous and extended enquiry, and interact with practising scientists.
Using an approach to materials that utilise science issues from the news into fully realised and resourced lessons that help teachers use often unfamiliar teaching approaches was based on an already popular model tried and tested resource lead approach lead in the UK.
The programme is supported by three key strategies that take a funnel approach to involvement, with more teachers at the ADOPT phase and fewer reaching the TRANSFORM phase, but those that do having a deeper involvement.
1) The first step, ADOPT, has a very accessible and attractive entry point to achieve take-up on a large scale.
2) For teachers who were curious to find out why the lessons worked, we developed workshops and online courses to teach the tools for using the materials effectively.
3) We set up an online community to stimulate reflective dialogue and interaction with more knowledgeable mentors.
Project Context and Objectives:
PROJECT CONTEXTS and OBJECTIVES
ENGAGE has delivered the majority its objectives during its 39 month course. It is clear from the individual partner reports that ENGAGE has had impact on science teachers, and also some influence on their students, also that it is leaving behind a strong legacy. It has been very challenging to develop and roll out three project stages, ADOPT, ADAPT and TRANSFORM in only three years, across 11 countries. However, both ADOPT and ADAPT achieved their objectives of recruiting large numbers of teachers to try ENGAGE materials, and then get them more committed to RRI by participating in professional development, or using ENGAGE to teach RRI skills as part of their teaching. As a result of the three-month extension, TRANSFORM was delivered in a limited form; however, there was not enough time to embed this fully, meaning that the numbers of teachers who could go through all three stages was limited.
Following the mid-term review in Brussels in December 2015 our project delivery was revised to be delivered within an extended 39 month project plan.
Of the components of ENGAGE, the most well-received by teachers have been the resource materials, with take-up and very positive reports from teachers. There was also some evidence that using the materials alone can shift practice towards RRI-based teaching. The professional development programme reached large numbers of teachers and was well received, and in some cases influenced future practice. The challenge was to get teachers who are under great pressure to deliver the curriculum in school to participate. Ability to tailor and localise CPD was key to success, so short workshops proved most effective in countries such as the UK. Longer online courses have worked very well in countries such as Spain where teachers want ongoing support. Setting up an online community provided the most difficult challenge. Few partners reported evidence of significant discussion between teachers online. Other countries tried to supplement the online community with face-to-face events, which have created a cadre of innovative teachers as part of the project legacy. In this project period, the focus shifted from engaging large numbers of teachers (ADOPT and ADAPT stages of the project) to working more closely and bespokely with individual teachers (the TRANSFORM stage) and collecting qualitative evidence of impact. All partners carried out case studies, and five went beyond this to collaborate on a ‘PCK research project’ to find out what teachers learned from using ENGAGE. (
Overall outputs
ADOPT outputs
We met and exceeded the target for curriculum materials, publishing 22 Topicals in all 11 countries. The total number of users registered in the ENGAGE knowledge Hub reached 16,107, with 10,218 of these having downloaded at least one ADOPT material (download). This number (of downloads) indicates a success rate of 87% on initial targets set in the DoW (see the WP4 report for more detailed information on targets). Evidence on usage of the ADOPT materials in the Knowledge Hub has been provided by more than 1,750 comments published by users. At least two workshops were conducted in each of the 11 countries of the consortium on the dilemma and group discussion teaching strategies, engaging a total of 1,268 teachers, almost double the target of 645 teachers set in the DoW. 25 online courses were delivered in the lifetime of the project in the 11 countries of the consortium, in which a total of 974 teachers enrolled.
It is difficult to estimate the number of students who have used the ENGAGE materials in school. However given the number of downloads and the positive responses to the materials it is estimated that the project will have come close to the targets as specified in the DoW.
ADAPT outputs
ADAPT achieved its aim of getting initial ADOPT users to get more involved in RRI-teaching. From a total of 11,940, 26% moved forward to ADOPT, so they could use the more advanced materials, exceeding targets. During months 19-39, 12 of these advanced materials were developed - ‘Sequences’. This was fewer than expected, mainly because it was a difficult challenge to create a design for longer materials that would actually get used by teachers in curriculum time. Also more time was put into localising ADAPT materials to partner preferences. However it is clear, from the number of downloads and comments, that these were successful. All Partners ran online courses, with 236 teachers completing: about 65% of the target. However this reflects the challenge of getting teachers to commit to non-essential activities that involve out-of-school commitment.
TRANSFORM outputs
For TRANSFORM, a lot of time was invested in creating a format for a ‘Project’ which teachers would consider of enough value to need several lessons. Two ‘Global Projects’ were created that were published across the consortium, and individual countries created their own ‘Local Projects’ based on locally-relevant issues. The total number of downloads for TRANSFORM Projects was more than 3000. Partners worked in a bespoke yet time-heavy way to individually support 42 teachers with their projects. About 150 teachers were involved in online courses, which is very close to the target. About half of the teachers completed the course.
The Three Phases of ENGAGE
Topicals (ADOPT): 22 of these ‘entry-level’ activities were developed, where students apply science knowledge and enquiry /RRI skills to a socio-scientific dilemma. This is more than the 20 we expected in the DoW. Topicals were very important to our strategy, and publishing them regularly throughout the project has enabled us to build a large user base.
Sequences (ADAPT): 12 of these ‘advanced level’ activities were developed, which explicitly teach students enquiry/RRI skills Sequences using games, and then apply these to solve a socio-scientific Dilemma. In the DoW, we expected to produce 20. The reason for developing fewer activities is that it proved much more complex and time-consuming than anticipated to create a Sequence that teachers would find sufficiently valuable to give 2 lessons of curriculum time for. Therefore we put more effort into creating a smaller number of Sequences to ensure their quality and uptake, which has been high.
Projects (TRANSFORM): A total of 16 extended enquiries into socio-scientific issues were developed, to enable students to work more autonomously using RRI skills. Two of these ‘global projects’ were developed by the WP3 team at SHU, to act as exemplars. These were translated by each partner. Additionally, each partner developed one or two of their own projects: a ‘local project’ customised for an issue relevant to their country. In the DoW we expected to develop up to 20 projects. As with Sequences, we found that development of high quality, project-based learning that would be useful to teachers across all countries was a complex and time-intensive process. Therefore we decided to limit production to the numbers above. Downloads of the global projects have been high and comments very favourable, justifying the emphasis on quality over quantity.
Project Results:
1.0 Deliverable reports
ENGAGE produced a substantial library of deliverable reports, a number of which may have use beyond the immediate consortium members. While a substantial number of these cover consortium specific areas of work such as dissemination and networking plans for the consortium and its outputs, others may be of wider interest. For example Deliverable report D1.1: Responsible Research and Innovation (RRI) curriculum document set out the steps taken to develop the theoretical framework under which the project would be implemented by outlining the current picture of existing RRI curriculum in all participating countries. This aimed to identify barriers and opportunities for the accomplishment of the project’s objectives by carrying out a science curriculum analysis covering 11 European countries, namely: United Kingdom, Greece, Germany, France, Romania, Israel, Spain, Norway, Switzerland, Lithuania and Cyprus for 11-16 years old students. Deliverable reports also set out guidelines and frameworks for the production of project materials for teachers and students, aiming for a common approach that could be adapted for different national contexts particularly around the pedagogy that the project wanted to encourage, teaching strategies and style of student activities and achieving alignment between project objectives and the materials created (see deliverable reports D3.5 and D3.6). The project created an RRI curriculum which aimed explicitly to teach skills like argumentation, which are vital for engaging with socio-scientific issues.
2.0 Website
The project website or Knowledge Hub was at the core of the ENGAGE project. It consisted of a platform in nine languages for 11 countries, which has supported more than 13 000 registered users in carrying out 130 000 downloads of the ENGAGE materials. Each country translated the project website into their own language, as well slides and video about the project, and media repositories of photographs and images, as well as Blog posts. Each country also set up social media channels: YouTube, Facebook and Twitter.
It also provided the platform for delivery online CPD in Massive Online Open Communities (MOOCs - see section 4.1).
The website was constantly updated and improved, in order to keep pace with the project's developments and the consortium's needs. For example:
• sending a notification email to MOOC tutors upon a new user's registration to the course
• supporting MOOC authors to switch to participant's view of the course
• providing a special label for tutors answering discussions in forums
• granting tutors an easy access to the list of participants to the course
• providing a file upload facility for course participants: the Staff Graded Assignment (SGA) module allows tutors to get files from course participants (upon a specific request of assignment) and give them a grade. The grade contributes to the overall course grade as per the specified grading policy.
• supporting the generation of certificates for the courses
It was also adapted to be able to better report on user activity, such as the possibility to download information about all posts at once instead of only per single post; numbers of downloads for a single package, in addition to number of downloads per lesson; data about activity on posts (comments, rates, likes); grouping of downloads per country of downloaders.
A Tag cloud plug-in was set up to enhance the search facility and appeal of the sites. An Enhanced Video Library page allows a quick and direct access from the platform to videos on the national YouTube channel, grouped by Playlists. Videos can be seen directly from the project portal, or reaching YouTube in one click. More than 10 videos about pedagogical tools, materials, RRI experts and Scientists were developed centrally, with some partners also creating their own. X animations were also developed and translated for the websites.
In the second part of the project teachers were able to contribute to the Knowledge Hub by adding details about their activities related to ENGAGE, with users being offered the chance to send their customisations of ENGAGE lessons and have them published on the relevant national site. For security reasons end users could not be allowed to publish posts directly, a form was provided on the website we proposed and discussed with partners two possible policies: providing a form to be completed by users Data are evaluated by the related national site owner and approved lessons are published on the site into a dedicated gallery.
A Partnership Brokering System (PBS) for each national site was set up using the Knowledge Hub as its base. The aim of this system was to support schools to find and select relevant ‘RRI experts’, and created scheduled mentoring partnerships for carrying out the RRI projects which are part of the Transform step of phase 2 deployment. The PBS is implemented as a directory of listings of experts from different areas. The directory is organised in six categories (types of experts): Expert RRI Teachers / Mentors; Media Experts; Science Museum Representatives; Industry Stakeholders; Researchers / Scientists, and Tutors. The main functions of the PBS are: (a) browse the PBS content (lists of experts and categories); (b) view the information of a specific expert or all the experts of the same type; (c) send messages to a specific expert; (d) introduce in the database a new expert and information about him; (e) simple and advanced search for a specific expert.
Learning analytic tools were used for collecting data about students’ opinions, values and knowledge before and after using ENGAGE curriculum resources, for formative and summative evaluation. This was done by embedding quizzes and short surveys within the activities of the curriculum resources. Learning Analytics were used to collect data about teachers’ activities in the ENGAGE Knowledge HUB. Several tools were developed and implemented to collect quantitative and qualitative data about teachers’ behaviour on using resources of ENGAGE project. Data were also collected using external tools (Google Analytics, AWStats), but also by developing and installing supplementary WordPress plugins. These tools provide number of users; numbers of downloads; users statistics by location; new visitors; most downloaded lessons etc. In order to collect qualitative data, quizzes and surveys were developed that were applied to teachers before, during and after different activities (on-line courses, promoting teachers in a new ENGAGE stage etc.).
The ENGAGE Knowledge HUB can be seen as a community of interest where science teachers can learn about RRI from other teachers and experts; they are able to review and explore the materials that other teachers are using and they can propose new strategies for implementation. The reflective comments on the materials and their usage can be found both on the relevant web page, but also in the forum discussions.
Using the data collected by Learning Analytics tools, every partner was able to monitor their local teaching community. The numbers were reported periodically, centralised, and discussed in online and face-to-face meetings. When circumstances required, partners proposed actions and they took measures to guide the local ENGAGE teachers’ community to fulfil the project targets.
3.0 ENGAGE Materials
A central goal of the ENGAGE project was to develop high quality curriculum materials and online course content for the ADOPT, ADAPT and TRANSFORM stages, integrating RRI with the ‘scientific inquiry’ skills that are common to almost all countries’ curricula. The purpose of the three different types of materials was to give a stepped approach to inquiry by introducing teachers to the project principles of RRI with a short ADOPT classroom activity, to then encourage their confidence and progress to the longer ADAPT materials and then on to the longer project based TRANSFORM work with their pupils. ENGAGE prioritised a set of 10 processes which became the learning objectives of our curriculum materials:
• Define questions
• Analyse patterns
• Draw conclusions
• Communicate ideas
• Justify opinions
• Critique claims
• Interrogate sources
• Use ethics
• Estimate risks
• Examine consequences
A clear unique selling point of these materials was the fast track to online publishing of 4-6 weeks, meaning that the materials were very current and the issues involved still fresh in students’ minds when taught in class. This was achieved with a tight development workflow with several stages of tightly scheduled input and consultation from other partners. Localisation of the materials was carried out to meet the needs of each country. Each partner contributed information to make the materials fit their context, and altered the text as appropriate during the translation process. As a result the materials are one of the highlights of the ENGAGE project, in terms of their quality, their dissemination and uptake, and their impact.
ADOPT Topicals - Twenty-two of these ‘entry-level’ activities where students are able to apply science knowledge and enquiry /RRI skills to a socio-scientific issue and then given a stimulating activity for students to review the science and develop an informed opinion. This lesson model was developed hand-in-hand with two Adopt Tools: Dilemma Lessons and Group Discussion (see section XX). Through this integration, teachers using the materials would develop curiosity in how and why the lessons work, and be motivated to enrol in an online course or workshop. Topicals were very important to the ENGAGE strategy, and regular publishing of these throughout the project has enabled the building of large user base. As lessons these would last for one 40-60 minute session.
Examples:
Text neck - New research suggests that smart phone use is seriously damaging our necks. Should use smart phone less to prevent neck damage?
Eat Insects - Farming large animals uses precious resources. Can you persuade people to swap meat for insects?
Invasions - Common ragweed is an invasive plant which is spreading across Europe. Should we control it by introducing non-native beetles?
ADAPT Sequences - Twelve of these ‘advanced level’ activities were developed to explicitly teach students enquiry/RRI skills Sequences using games, which can then be applied to solve a socio-scientific Dilemma. These lasted from 1-2 lessons.
Examples:
Electronic cigarettes - Campaigners in support of ban point out that nicotine from e-cigarettes may contribute to heart disease and cancers, as well as damaging the brains of developing foetuses. In this activity students decide whether they support a ban. They apply their knowledge of particle theory to decide whether exhaled nicotine can reach non-vapers nearby, and then learn to judge risks to decide whether the benefits of a ban on indoor vaping outweigh the risks.
Two degrees - UK winters are getting wetter and flooding is a common threat in many parts of the country. Scientists believe climate change may have caused this extreme weather. In this sequence students apply their knowledge to create an apocalyptic weather report. Then they learn the skill of examining consequences, and judge solutions for limiting the temperature rise to 2 degrees.
TRANSFORM Projects - A total of 16 extended enquiries into socio-scientific issues were developed, to enable students to work more autonomously using RRI skills. These were translated by each partner. Additionally, each partner developed one or two of their own projects: a ‘local project’ customised for an issue relevant to their country. Downloads of the global projects have been high and comments very favourable, justifying the emphasis on quality over quantity. These projects were designed to take place over the course of several lessons. Transform activities allowed partners to share valuable experience in innovative teaching practices. A gallery of “Transform-like local projects” on the Knowledge Hub illustrated a very high degree of creativity across the consortium, and the benefits of sharing practices.
Examples:
Exterminate: Mosquitoes are the world’s most dangerous killer. The diseases they transmit; malaria, Zika and dengue fever, cause more than a million deaths per year. Some scientists have suggested exterminate all dangerous mosquito species. One method is to release genetically modified (GM) male mosquitoes which prevent further breeding. In this project, students investigate whether exterminating mosquitoes is a good idea, using scientific knowledge about interdependence.
Eco-phone - As the number of smartphone users worldwide exceeds 2 billion, and as users update their devices with ever-increasing frequency, there are growing concerns about the impacts of smartphone manufacture and disposal on the environment and human health. In this activity, students use knowledge about Earth resources, as well as applying their working scientifically skills, to work out how to persuade phone manufacturers to make eco-friendly smart phones from sustainable materials.
The feedback received about the materials has indicated they act as helpful exemplars for teachers unfamiliar with RRI teaching. However, in some countries where there is not a culture of using pre-prepared lessons, such as Greece and Cyprus, some teachers were less keen and want either to customise them, or improvise entirely new activities. This was encouraged by delivering materials in a format that allowed teachers to make their own changes and via teacher guides, workshops and courses to clearly state the project rationale, so teachers could make changes while keeping pedagogical principles intact.
3.1 Areas of commonality and localisation in Materials
The materials were centrally designed in an effort to have a common base between the materials used in all the partner countries. Since the main purpose was to design materials that promote RRI, all materials were designed based on a common framework that had been agreed within the consortium on what is defined as RRI, and the different types of skills that are considered important in the teaching of science. More specifically, the following aspects of the materials are common:
● Emerging science and technology contexts
● RRI goals
● Structure and design of materials
● Core Science Ideas
Despite the decision to have a common structure and design for the Materials, it was considered important to localise the Materials to meet the individual needs of each country and their science curricula. The main reasons for localisation:
● An initial review of the curriculum of the partner countries demonstrated different content and emphasis on different skills.
● Research has shown that when the materials are relevant to students’ lives, science is more engaging. Therefore the effort during localisation is to use information and data from each partner country to make the materials relevant.
● RRI includes ethical and moral aspects, and these aspects are different in each country.
Aspects of materials which are localised:
● Specific contextual information (e.g. information about the sources of energy used in each country)
● Scientific data (e.g. data about consumption of energy in each country)
● Science curricula links (each partner provides the links to the national curriculum)
● Additional content (e.g. according to the national curriculum there might be links to further knowledge)
● Web links for news stories and media (e.g. news stories from the local press from each country)
3.2 Beyond the ENGAGE consortium
Use of ENGAGE materials has spread to 80 countries, based on data of downloads from the website. In Vietnam, China, Peru, Columbia and Uruguay, we have received direct reports of schools taking up ENGAGE and embedding into their curriculum. The largest impact has been in Brazil, where 4 different cities ran very large projects with schools, using the ‘Zika’ ENGAGE material which is so relevant in Brazil. Around 80 teachers and more than 1000 students participated in workshops, and students presented posters at a national exhibition. Equally important, the students collaborated with 36 scientists and experts to develop their understanding of RRI.
4.0 Continuing Professional Development (CPD)
ENGAGE used a combination of face to face workshops and on-line course and webinars in order to deliver effective Continuing Professional Development to teachers. It was evident that a rigorous professional development framework was needed to inform the development of the CPD workshops covering guiding principles, strategies, prototypes and pedagogical design. The framework describes each element of our PD programme, and provides a clear objective and rationale for the strategies chosen for ENGAGE. Second, for teachers who were curious to find out why the lessons worked, we developed workshops and online courses to teach the tools for using the materials effectively
Important outcomes of the workshops included;
• familiarisation of teachers with dilemma and small group discussion RRI teaching strategies;
• hands-on engagement with the Adopt materials;
• motivation to participate and engage in the on-line courses for more in-depth understanding and knowledge co-sharing.
The starting point was to describe what ‘practices’ were needed from ENGAGE teachers at the end of the project;
• Use authentic tasks to help students apply science learning to every-day life
• Explicitly teach the ‘RRI skills and knowledge’ needed to deal with science issues
• Use open dialogue to build students’ reasoning and understanding
The latter of these is probably the hardest practice to implement involving a big shift in interaction style from one where the teacher’s views and ‘correct answers’ are dominant, to one where the students’ ideas and argument are more important.
CPD tends to be more effective when it focuses on well-defined skills rather than general practices; therefore ENGAGE followed a similar approach to defining the skills as used in the ‘Ambitious Science Teaching’ project and the FP7 inquiry project ‘TEMI’. This aims to turn the practices into a small number of easy to use ‘Tools’ for teaching. Six tools were developed - Dilemma Lessons, Problem-solving Sequences, Scenario-based Topics, Group Discussions, Class Conversations, and Performance Assessment. Instructing and coaching teachers in the use of the Tools in the classroom was the focus of the CPD ENGAGE programme.
Two CPD workshops were conducted in each of the 11 countries of the consortium, on the dilemma and group discussion teaching strategies, engaging a total of 1268 teachers. In addition partners also rolled out dozens of workshops that engaged more than 1200 teachers.
In addition in Cyprus, Romania, Greece and the Netherlands, partners have managed to incorporate ENGAGE into both undergraduate and post-graduate initial teacher training, and Masters courses for teaching and science communication. This includes using specific ENGAGE lessons and ‘Tools’ for RRI-based teaching within the course programme, so that many new teachers will be exposed to ENGAGE at the beginning of their careers, and are likely to integrate our approaches into their repertoire.
4.1 Massive Online Open Communities (MOOCs)
Based on the professional development framework above and in addition to the workshops, online courses were created for each project stage (ADOPT, ADAPT and TRANSFORM). Twenty five of these courses were delivered across 11 countries, with 974 teachers enrolled in total.
The online courses in the partner countries were based on common content but differed between contexts in terms of the structure, the mode of delivery and the facilitation. This locally relevant approach to delivery was decided upon in order to make courses most relevant to different national contexts and teachers attitudes. In the UK for example methods used for the delivery included forum posting, review questions, assignment activity and webinars. The Greek and the Spanish online courses, on the other hand, followed a more structured and rather traditional mode of delivery, with content, tasks and assignments being uploaded each week by the course facilitators. In the Spanish courses, each teacher created their padlet (https://padlet.com/) to upload lesson plans with ENGAGE materials or RRI CPD tools, as well as photos from the lesson, student products and lesson reflections. Each teacher had to comment on the padlet from another teacher as part of the activities to complete the course. Facilitators commented on the padlets as well. The above are examples of how the courses in different cases supported interaction with other teachers and facilitators.
The success of these courses varied across the consortium. In some contexts the courses were very successful in getting teachers to use the materials, and reflect on the pedagogies of teaching with RRI. However, in others, it proved difficult to recruit many teachers and have them complete the course. Around two thirds of teachers participating reported that they learned strategies for developing students’ inquiry skills to a high extent, and half the teachers said they learned how to structure effective discussions.
5.0 RRI festivals
Eleven countries organised a final ENGAGE event, allowing teachers to meet, present results of local Transform projects, discuss open schooling and innovative pedagogies, as well as RRI topics related to science education.
Andy - need more on these???
6.0 Internal evaluation
The internal evaluation of ENGAGE focussed on the impact of the project's CPD activities and innovative teaching materials on teachers as the core target group, while also carrying out a more limited impact assessment for students given the drive to improve their knowledge and skills via teacher involvement. The evaluation included both quantitative and qualitative data and was driven from within the consortium.
A pre and post ENGAGE CPD questionnaire was used to examine 56 teachers' responses (from five countries - Romania, Cyprus, Lithuania, UK, and Spain ) against the five dimensions of RRI teaching established by the project:
• Teachers’ Knowledge of RRI
• Teachers’ conception of their own role
• Teachers’ use of discourse
• Teachers’ conceptions of learning goals
• The nature of classroom activities
This established that there was a positive shift in all five dimensions in the sample of teachers. In particular there was only a statistically significant positive shift seen in Dimension 1 "Knowledge of RRI".
In terms of qualitative research with teachers a descriptive qualitative study was designed to investigate the change in teachers’ Pedagogical Content Knowledge. This used three instruments - a Lesson preparation form (assessing PCK before use of ENGAGE material/workshop participation), Lesson reflection form (assessing PCK after use of ENGAGE material/workshop participation) and an Observation table to support PCK data collection, which were refined after a pilot study was carried out.
Thirty one teachers from four partner countries (Spain, Norway, Israel and Cyprus) participated in the PCK study. The results are summarised in Table 1 below.
Table 1. Summary Results of teachers’ PCK using ENGAGE materials in five countries
Country (N-teachers)
Goals in line with ENGAGE
objectives
Focused mainly on skills
Focused mainly on science
Aware of
specific student difficulties
Adapted the material to fit their students
Prepared assessment activity /
tool
Changed PCK (main
aspects*)
Spain (7)
5 teachers
1
1
3 (misc., beliefs)
3
1
7 (IS, US)
Norway (7)
3 teachers
2
2
6 (skills, beliefs)
1
0
3 (IS)
Israel (10)
4 teachers
6
0
5 (skills, beliefs)
2
0
10 (IS)
Cyprus (7)
6 teachers
1
0
6 (misc., beliefs)
6
0
4 (IS, US)
*IS =Instructional strategies, US= students’ understanding
As illustrated, 18 teachers articulated goals that were in line with ENGAGE objectives (i.e. students apply science to new contexts, use higher order thinking to form evidence-based opinions on societal needs and values). It is interesting to note that the analyses showed that years of general teaching experience had no direct correlation with the quality of the teachers’ PCK. The main conclusion is that by using these ENGAGE materials some of the teachers changed aspects of their PCK related to ‘Instructional Strategies’, and ‘students' understanding of science’. Analysis of individual teachers’ profiles suggests that these changes are linked to teachers reflecting on specific difficulties they faced during teaching - in particular difficulties linked to coordinating argumentation activities, group work and discussing uncertainty. Moreover, in all countries, the less-developed PCK component was about ‘Assessment’. This might be linked to the fact that most of the teachers implemented the ENGAGE lesson as a separate lesson, not linked to their curriculum. This may suggest that PCK can shift by using educative materials, without necessarily engaging in long-term professional development.
The way of teaching science
We conducted an evaluation survey of users of ENGAGE materials, across all partner countries. 545 teachers responded. The majority (44%) of the teachers reported that the ENGAGE materials enriched their way of science teaching ‘above the average’, and the second biggest (30%) group of the teachers reported this as to a ‘great extent’.
In-depth case studies
Each partner carried out one or more case studies to focus on teachers who had used ENGAGE. Overall, from the case studies, it was found that ENGAGE materials enabled teachers to think creatively about how to teach science. Some teachers also learned to teach by starting with a good question, making the students think by themselves and get them to find a solution to the Dilemma.
Online Course (MOOC) impact
A total of 297 teachers took part in the evaluation of the online courses, the results of which are given in Table 2.
Table 2 - results of MOOC evaluation
Teacher learning/skill development
% agreeing
learned strategies for developing their students’ inquiry skills through ENGAGE online courses at a high extent
55%
mastering the use of 5E lessons to develop RRI/enquiry skills
52%
learned how contemporary science can engage students and get them thinking and talking through ENGAGE online courses at a high extent
53%
understood how to use ‘student’ thinking guides to scaffold RRI/problem-solving skills
51%
learned how to prepare students for effective discussion by building an argument based on evidence and reasoning
51%
effective discussion using different components of argumentation and designing their own problem-based lessons with argumentative conversation for RRI/enquiry skills
49%
learned to assess when inquiry or working scientifically could be used within the curriculum
44%
Potential Impact:
Please provide a description of the potential impact (including the socio-economic impact and the wider societal implications of the project so far) and the main dissemination activities and the exploitation of results. The length of this part cannot exceed 10 pages.
The Impact and legacy of ENGAGE
Lots of ENGAGE users in some countries
The extent of take-up of ENGAGE materials has been substantial. In Lithuania, 16% of all science teachers are estimated to have used ENGAGE. In the UK, a third of all science teachers have registered and more than 10% have so far used the materials, based on survey data. These numbers indicate that there are a good number of teachers who, because of their positive experiences, are likely to spread ENGAGE by word of mouth so that the number of users continues to rise.
Effective online courses and workshops
The ENGAGE online courses were a localised success in countries such as Spain, Greece, Israel and Cyprus, getting teachers to use the materials, and to reflect on the pedagogies of teaching with RRI. However, in other countries it proved difficult to replicate the success, often falling down at the recruitment of teacher stage. Around two-thirds of teachers participating reported that they learned strategies for developing students’ inquiry skills to a high extent, and half the teachers said they learned how to structure effective discussions. In Greece, in-depth evaluation found a clear positive impact on raising teachers’ awareness of RRI in science education. In some countries, workshops were an easier way to get teachers to participate in formal professional development. The internal evaluation found these were also influential in helping teachers gain confidence in using group discussions and dilemmas in science lessons.
Evidence of a shift towards RRI-based teaching
The internal evaluation, carried out by Tu Delft, found evidence that teachers shifted their practice towards RRI-based teaching as a result of involvement in ENGAGE. Using a 5-dimensional model to characterise this shift, and from the statistical analysis, there was strong evidence that ENGAGE could improve teachers’ knowledge of RRI. In the other dimensions, such as teachers’ roles and their goals for science teaching, there were only slight increases in the mean values. With limited time to implement the TRANSFORM stage most of the evidence was captured from ADAPT teachers. They had modest experience of ENGAGE compared to the TRANSFORM teachers who had been through the whole programme. A more significant shift in RRI-based teaching may have been found if more TRANSFORM teachers had been surveyed.
In the research into teachers’ pedagogical content knowledge (PCK), it was found that using ENGAGE materials enabled some teachers to successfully develop new instructional strategies for addressing socio-scientific issues. The internal evaluator concluded that ENGAGE’s materials were sufficiently educative for teachers to learn new strategies for RRI-based teaching without engaging in formal professional development.
Increased student motivation and inquiry skills
The thousands of comments on the website provide anecdotal evidence ENGAGE makes science lessons highly motivating for students. Our external evaluator found some evidence of a positive influence on student ENGAGEment from video recordings of two teachers’ lessons. When students were experiencing an ENGAGE lesson, their ‘time on task’ seemed to improve compared to before the lesson. In Romania, the impact of ENGAGE on students’ behaviour was studied with 64 students. Anecdotal evidence suggested that after using ENGAGE the majority of students began to read and discuss things related to science and technology more outside of classroom. Each partner also carried out case studies on their teachers’ use of ENGAGE, and in all cases using ENGAGE seemed to improve students’ inquiry skills. However, in terms of other stakeholders such as scientists, the evaluation was unable to show impact.
Continuing usage of ENGAGE materials
The ENGAGE website and its collection of curriculum materials is a major legacy for the project. It will be maintained for three years, with 37 high quality curriculum materials in 11 languages to help teachers use socio-scientific issues with their students to develop RRI skills. The materials are backed up by articles and discussion papers for teachers to deepen their knowledge of RRI. In most partner countries, the online course content will continue to be accessible for teachers to participate in a localised way. Based on the 16,000 teachers already registered, and word of mouth, it is hoped that the numbers of downloads, usage and comments will continue, even without partners’ direct involvement.
Impact on science curriculum policy
ENGAGE was particularly successful in influencing curriculum policy in the UK, working with the largest awarding body (AQA) to incorporate ENGAGE’s ‘RRI curriculum’ into its Science Syllabus for 11-14 year olds. The Syllabus promotes ENGAGE activities as ideal resources towards meeting one of the assessment objectives - to apply knowledge. AQA and other publishers such as Hodder have specifically recommended ENGAGE on their websites, and this will ensure that many teachers use the materials as a core part of the curriculum for several years.
Embedding into teacher training
In Cyprus, Romania, Greece Lithuania, UK, and the Netherlands, partners have managed to incorporate ENGAGE into both undergraduate and post-graduate initial teacher training, and Masters courses for teaching and science communication. This includes using specific ENGAGE lessons and ‘Tools’ for RRI-based teaching within the course programme, so that many new teachers will be exposed to ENGAGE at the beginning of their careers and are likely to integrate the approaches into their teaching.
ENGAGE beyond consortium countries
Based on data of downloads from the website, the use of ENGAGE materials has already spread to 80 countries. In Vietnam, China, Peru, Columbia and Uruguay, there are direct reports of schools taking up ENGAGE and embedding into their curriculum. The largest impact has been in Brazil, where four different cities ran very large projects with schools, using the ‘Zika’ ENGAGE materials which are of obvious relevance in Brazil. Around 80 teachers and more than 1000 students participated in workshops, and students presented posters at a national exhibition. Equally importantly, the students collaborated with 36 scientists and experts to develop their understanding of RRI.
ENGAGE further potential Impact
Below are a series of promising examples of different aspects of ENGAGE that offer future potential in terms of offering potential for localised legacy. We would rate these as all tentative yet positive aspects with future value to them. Including
Website – Romania, France, Germany, Lithuania, Switzerland, Israel, Norway, Greece, Cyprus, Spain and the UK.
Changed teacher Pedagogy – The Romanian experience
A key achievement of ENGAGE is the high level of Engagement of the Romania partner with government officials at different levels of the education infrastructure which ensures they play a major role in discussions about science curricula and teacher training.
As stated in the DOW, the project expected at least 2 pre-service institutions and 2 in-service training providers to have integrated elements of the ENGAGE programme in their pre/in-service training, within each country.
Partners have worked with teacher training institutions to embed exposure to and use of RRI pedagogy through ENGAGE OER and guidelines for teaching. The degree of impact has varied across partners. The evidence suggests that a number of factors account for this, but three key factors are the expertise of the partners, the institutions they are located in, and their role within these institutions. This configuration of factors was not evident with all partners.
A prime example of achievement that is being further developed is Romania. The partner works within a university that provides teacher training. A key role its three members have played for several years is in the development and delivery of pre and in-service teacher training. The Romanian partner achieved wider penetration across education levels, disciplines, and across the country than may have been expected.
First, the partner was able to integrate ENGAGE materials and RRI techniques into the curriculum at undergraduate and postgraduate levels and provide a number of sessions for pre and in-service teachers. Second, the ENGAGE project was focused on secondary education but the partner was able to integrate RRI pedagogy and some elements of the OERs into teacher training for teachers of pre-school and primary levels. Third, ENGAGE focused on science education. The partner was able to include RRI pedagogy and some elements of the OERs into a range of disciplines such as Maths and other programmes in the Faculty of Orthodox Theology as well as in programmes in Geography, Food Processing, Economics, Technological Education, Education for Health, Counseling, Career Orientation, Religion and Music.
Finally, there is potential for wide geographical reach as there is a very high probability that this new approach will influence the development of teacher training across Romania. Discussions with Inspectors of Science subjects at county level, and with government officials working in national teacher training development programmes and national education policy levels, testifies to this. The partner and other colleagues at the university have a strong pre-existing relationship with policy officials and professional teacher associations. The views of the partner are respected and they were involved in frequent bilateral and multi-lateral meetings concerning teacher education. As teacher education is very tightly controlled in Romania with teachers having to undertake nationally accredited training to both obtain and maintain their teacher status, the fact that the partner has a seat at the table, both county and national, and is able to draw on its experience of delivering ENGAGE, is an excellent achievement for the project.
The School Science Curriculum : The UK experience
The AQA KS3 science curriculum has been developed in conjunction with the enquiry aspects of Enquire, Analyse, Communicate and Solve all have a strong link to the ENGAGE model.
The UK partner has had a significant and major impact on the science education curriculum in the UK. It has ensured that the ENGAGE framework for science teaching has been adopted by one of the UK national exam boards, AQA. While not the only exam board in UK, the AQA is a major board. Both the mobilisation of existing contacts and serendipity played a role in making this happen.
The new Key Stage 3 (for 14-16 year olds) Science Syllabus from AQA in the UK shifts the focus of science education from the acquisition of knowledge. Jointly developed by ENGAGE partners, its breakthrough design reduces the science knowledge content teachers need to cover, and shows them how to put science in context by giving students the skills to apply their knowledge to real life. It is hoped that this will help make learning science an exciting experience which will inspire current and future generations of students.
The UK partner has been able to use the AQA branding on its website and to publicise and run its events. At the same time, the AQA website provides additional credibility by links on its official website to the ENGAGE Knowledge Hub.
At the RRI festival that the UK partner had in March 2017, the Head of the AQA Science section gave the keynote speech. This link between curriculum and ENGAGE is a powerful one that will give momentum to ENGAGE resource uptake in the UK going forwards.
Website and knowledge hub – All partner and beyond experience
Work Package 2 lead by the SME Lattanzio learning partner No based in Genoa Italy. The Lattanzio learning team were well supported in their website realisations by the team at SHU and OU. Our materials are accessible and available through 11 different languages. In 2017 ENGAGE won a worldwide Open Education award.
A major achievement of the project is the Knowledge Hub, located on an attractive, user-friendly website. This houses an extensive range of Open Educational Resources (OERs) in English and the nine other languages of partner countries. One OER has been translated into Portuguese and another into Arabic. It has won international recognition through a prestigious award. The Knowledge Hub has around 18,000 registered members, but well over half are from UK. There have been approximately 130,000 downloads, and although all the partner countries have had their resource materials downloaded, the majority are downloaded from the English language webpage. Teachers from 85 countries have accessed the website demonstrating the wide reach of the project. There has been a high degree of exploitation of the OERs, which have been widely disseminated though digital and print channels, and the profile of the project has been raised through conference presentations and publications. While there is no robust evidence collected by the project to confirm that the target of 11,750 teachers using the materials has been reached, a guesstimate based on the number of registered users and downloads suggests that the target may well have been met. The website will be maintained until 2020 and there is a strong likelihood that the web traffic will continue to rise. The functionality of the website is high and offers potential as its online presence will be supported for the following 3 years. The website should act as a driver for future CPD activity and further teacher Engagement in some countries will continue to grow through word of mouth by teacher users recommending it to other teachers.
The Lattanzio group (ELS, Italy) provided Knowledge developed the Knowledge Hub (KH), namely the technical platform supporting the project’s main activities. The work focused on the development of the project website, the materials platform and online community. It also included the development of the virtual environment for the online courses and the learning analytic tools to record the use of materials and online community interactions. Complementing the work undertaken by the Lattanzio group was VUT (Romania) and the OU were also involved in the development of the EdX platform and analysis of technical aspects in order to assure a secure and trusted on-line environment for the KH.
It is commendable that the Lattanzio group paid considerable attention to designing the website and continuous improvements have been made to it throughout the project to increase its functionality and user friendliness. As they pointed out:
From our side, we tried to be as supportive as possible. When partners asked for technical assistance, we provided user guides and video tutorials, showing the procedures step by step and offering to assist live in Skype via screen sharing. (ELS Italy)
The KH has been enriched by new technical functionalities which have promoted more user-generated content as follows: (a) the rating system allows new users to identify appropriate materials more easily (b) teacher ability to respond to other comments has facilitated the development of reflective conversations (c) the personalization function reminds users to make comments on the last downloaded materials and the comments made by an expert teacher can be highlighted by adding the ‘Expert’ tag and (d) the brokering system contains the list of all experts, presents the expert profile and provides a way to establish contact between users and a specific expert. Partners could monitor progress and acquire analytical data on a regular basis which enabled them to make changes to improve accessibility to materials and promote better interaction with users.
Despite the availability of the improved functionality, the full potential has not yet been realised by all partners and users.
Use of the Knowledge Hub by Teachers outside the Partner Countries
The availability of OERs in ten languages increases its accessibility to beyond the English-speaking world. Most web traffic is evident on the English language page which reflects the importance of English as an international language and medium of instruction in many countries. The users of the website come from around 85 countries including Nigeria, Indonesia, China, India and Brazil. Some of the teachers from these countries who have accessed the website are local people, while many others are foreigners with English names which suggest they may be British teachers working overseas. Some of the teachers have downloaded the materials but others have merely registered as users.
One of the intended legacies of the project is use of its resources by teachers worldwide. While the geographical reach achieved is in itself a notable achievement, greater impact could have been achieved by engaging with these foreign and local teachers to help them use the material and also to mobilise them to act as country hubs. An attempt in this direction has been made. The most commendable, and likely to have sustained impact, is the work that ENGAGE has done with Brazil. The Zika resource material as with others such as 2 degrees show the value of rich context based contemporary resource materials to generate purposeful teacher and student discussion whilst developing RRI skills.
A key learning outcome for the co-ordinating partner (SHU) and project management coordinator (Andy Bullough) has been that centrally developed resources can work really well across delivery partners in different countries as long as they offer the dissemination partners the opportunity to localize and personalize the materials to fit within their own national contexts. This international project is the result of the laudable mobilisation of the contacts in Brazil of an ENGAGE team member working at the Open University.
The conceptualisation of this project and its scale went way beyond anything attempted with the ENGAGE project. The project ran for a month and used 36 Brazilian research educators as well as teachers and scientists. There were 1,473 learners including deaf people, older citizens, secondary and primary schools and members of local communities.
The project with Brazil investigated how ENGAGE materials could be easily disseminated to promote inquiry skills for RRI in different areas of Brazil and promote collaborations between universities and schools. Participants used the ENGAGE ‘GM decisions’ and ‘risk’ games to develop informed based opinions about genetic modified food through ‘Open schooling’. This means schools working in cooperation with researchers, parents and experts, and becoming an agent for community well-being. Families were encouraged to become real collaborators by interacting through social media and events. Biotechnologists and agro biodiversity consultants working on GM and wider society were also involved in discussing real-life projects in the classroom.
The Brazilian educators found that the ‘GM decisions’ and the ‘risk’ game could be easily embedded in the Brazilian curriculum but it required proper planning. Students interacted with researchers and scientists and co-created various examples to communicate their results. The project’s achievements included the creation of various products: 1 exhibition, 9 games, 4 new OERs, 42 illustrations, 1 webinar, 28 concept-maps, 1 sign-language activity for deaf people, 2 posters, 2 workshops and 4 video clips. The project concluded that inquiry skills for RRI can be fostered in many Brazilian states.
The work with Brazil has been a very good outcome for the ENGAGE project. It is likely that many of the OERs will be translated into Brazilian Portuguese for use by teachers all over Brazil. Indeed, the OER called Exterminate has already been translated and sits on the English webpage, as a webpage in Portuguese does not exist. The OER is very relevant to Brazil as it focuses on mosquitoes and many of the diseases they transmit, such as Zika, are rife in Brazil. The teachers and other stakeholders involved in the Brazil project have the potential to form valuable and sustainable hubs supporting communities of practice.
Another small but laudable attempt to make ENGAGE materials available to speakers of other than the ten languages on the KH is the case of Israel. The webpage of Israel, which is in Hebrew, also includes one OER, Ban Cola, translated into Arabic. This will be very valuable for Arabic-speaking teachers who have attended events organised by the Israel partner but have not been able to follow up with their Arabic-speaking students because they did not have resources in the language. It would be very helpful if funds could be obtained to enable all the materials to be translated into Arabic. This would be of benefit not only to the Arabic speakers in Israel and Palestine, but also for the hundreds of Arabic-speaking migrants who live in the partner countries.
The statistics on the website on the large number of registered users indicates a community of interest. Even if they do not download the materials or leave comments, a large number of teachers and science educators from the partner countries, supplemented by a few from a wide range of around 85 countries, have been exposed to the project and to RRI teaching. Some teachers who have not used the materials may, nevertheless, have used the RRI pedagogic approach and applied it where they could to their existing teaching practice. For instance, they may have involved their students through using the idea of a dilemma. Hence, the impact on some teachers has been of the RRI pedagogic approach, whereas for other teachers, they have not only adopted the pedagogic approach but also used the OER materials.
The Lattanzio group will support the maintenance of the website until 2020. It is anticipated that the web traffic will increase over time. Some indication of this is evident from the fact that in February 2017, 15,000 users were registered on the KH but towards the end of March 2017 this had risen to 18,000 – an increase of 3,000 in about a month. There are precedents for this pattern of increased traffic. Two examples are given here.
A project in which the ENGAGE PD was called science upd8 – a forerunner to ENGAGE - had continued web traffic of about 5,000 visitors per month for 5 years and there were downloads of the materials after the development finished. The ENGAGE project co-ordinator was involved in 2007-2010 in a project called cre8ate maths. According to him, the cre8ate maths website had approximately 2,000 users at the project end in 2010. This number increased through word of mouth to nearly 6,000 users (signed up members downloading content) by 2012m and as recently as 2015, still had 4,000 active users. Since the end of the project in 2010 no updating of the website has taken place.
It will be interesting to see if a similar trend will be manifested on the ENGAGE KH. There are some steps being taken to stimulate the growth of users. For instance, to increase the impact on countries worldwide, the UK partner is planning to work with the British Council in UK and with their multiple local offices worldwide, to increase the publicity and use of the KH. Another step, which all partners have undertaken, has been to disseminate their individual language based resources widely to a range of online libraries and repositories. Furthermore, each partner can load the resources onto the website of their own institutions. There have also raised the profile of ENGAGE through conference presentations and publications and many partners expect to continue with these activities. These wide dissemination activities, while noteworthy in their own right, will also draw teachers to the KH where they can make contact with other users, read and leave comments, gain ideas about how to adapt the lessons for their own students and access experts.
If the potential of the KH is realised over the next few years, then with its high quality open education resources and multiple functionalities, it is is likely to be one of the most enduring and significant achievements of the ENGAGE project.
List of Websites:
http://www.engagingscience.eu/en
Shifting school science towards RRI
ENGAGE aimed to give the next generation of students the knowledge, skills and attitudes to deal with socio-scientific issues in their lives, and develop informed opinions on emerging science and technology. ENGAGE’s goal is to embed Responsible Research and Innovation (RRI) within the science curriculum and change the way science is taught. Through a set of innovative strategies, we have attempted to shift the emphasis from transmitting a body of scientific knowledge towards applying science to issues that matter to students. At the heart of ENGAGE is using authentic activities to simulate how citizens conduct inquiries. ENGAGE operates on a major scale. More than 15,000 teachers have signed up across 11 partner countries: UK. Greece, Germany, France, Romania, Israel, Spain, Norway, Switzerland, Lithuania, and Cyprus.
State of the art innovation
Teachers need make major changes in practice to use socio-scientific issues in the classroom. Therefore ENGAGE chose a thought-out, multi-pronged approach and created a series of positive conditions for change. At the heart of the project were high-quality lessons that would encourage RRI-based teaching through effective engagement. We also constructed an escalated pathway of progression in teaching skills load to foster teacher change- In the ENGAGE model, once teachers gain confidence with the intuitive ADOPT teaching approaches, a proportion progress to using more advanced ADAPT materials in the next stage. This second stage involves a deeper interest and commitment to the ENGAGE philosophy, because the materials demand more curriculum time to explicitly teach RRI skills. The third step, called TRANSFORM, is our experiment in project-based learning and an opportunity for students to develop their transferrable skills. It was designed for a small proportion of teachers who wanted to make RRI and socio-scientific issues a major focus of their curriculum enhancement. We provided support to help them plan an issue-based project where students could engage in more autonomous and extended enquiry, and interact with practising scientists.
Using an approach to materials that utilise science issues from the news into fully realised and resourced lessons that help teachers use often unfamiliar teaching approaches was based on an already popular model tried and tested resource lead approach lead in the UK.
The programme is supported by three key strategies that take a funnel approach to involvement, with more teachers at the ADOPT phase and fewer reaching the TRANSFORM phase, but those that do having a deeper involvement.
1) The first step, ADOPT, has a very accessible and attractive entry point to achieve take-up on a large scale.
2) For teachers who were curious to find out why the lessons worked, we developed workshops and online courses to teach the tools for using the materials effectively.
3) We set up an online community to stimulate reflective dialogue and interaction with more knowledgeable mentors.
Project Context and Objectives:
PROJECT CONTEXTS and OBJECTIVES
ENGAGE has delivered the majority its objectives during its 39 month course. It is clear from the individual partner reports that ENGAGE has had impact on science teachers, and also some influence on their students, also that it is leaving behind a strong legacy. It has been very challenging to develop and roll out three project stages, ADOPT, ADAPT and TRANSFORM in only three years, across 11 countries. However, both ADOPT and ADAPT achieved their objectives of recruiting large numbers of teachers to try ENGAGE materials, and then get them more committed to RRI by participating in professional development, or using ENGAGE to teach RRI skills as part of their teaching. As a result of the three-month extension, TRANSFORM was delivered in a limited form; however, there was not enough time to embed this fully, meaning that the numbers of teachers who could go through all three stages was limited.
Following the mid-term review in Brussels in December 2015 our project delivery was revised to be delivered within an extended 39 month project plan.
Of the components of ENGAGE, the most well-received by teachers have been the resource materials, with take-up and very positive reports from teachers. There was also some evidence that using the materials alone can shift practice towards RRI-based teaching. The professional development programme reached large numbers of teachers and was well received, and in some cases influenced future practice. The challenge was to get teachers who are under great pressure to deliver the curriculum in school to participate. Ability to tailor and localise CPD was key to success, so short workshops proved most effective in countries such as the UK. Longer online courses have worked very well in countries such as Spain where teachers want ongoing support. Setting up an online community provided the most difficult challenge. Few partners reported evidence of significant discussion between teachers online. Other countries tried to supplement the online community with face-to-face events, which have created a cadre of innovative teachers as part of the project legacy. In this project period, the focus shifted from engaging large numbers of teachers (ADOPT and ADAPT stages of the project) to working more closely and bespokely with individual teachers (the TRANSFORM stage) and collecting qualitative evidence of impact. All partners carried out case studies, and five went beyond this to collaborate on a ‘PCK research project’ to find out what teachers learned from using ENGAGE. (
Overall outputs
ADOPT outputs
We met and exceeded the target for curriculum materials, publishing 22 Topicals in all 11 countries. The total number of users registered in the ENGAGE knowledge Hub reached 16,107, with 10,218 of these having downloaded at least one ADOPT material (download). This number (of downloads) indicates a success rate of 87% on initial targets set in the DoW (see the WP4 report for more detailed information on targets). Evidence on usage of the ADOPT materials in the Knowledge Hub has been provided by more than 1,750 comments published by users. At least two workshops were conducted in each of the 11 countries of the consortium on the dilemma and group discussion teaching strategies, engaging a total of 1,268 teachers, almost double the target of 645 teachers set in the DoW. 25 online courses were delivered in the lifetime of the project in the 11 countries of the consortium, in which a total of 974 teachers enrolled.
It is difficult to estimate the number of students who have used the ENGAGE materials in school. However given the number of downloads and the positive responses to the materials it is estimated that the project will have come close to the targets as specified in the DoW.
ADAPT outputs
ADAPT achieved its aim of getting initial ADOPT users to get more involved in RRI-teaching. From a total of 11,940, 26% moved forward to ADOPT, so they could use the more advanced materials, exceeding targets. During months 19-39, 12 of these advanced materials were developed - ‘Sequences’. This was fewer than expected, mainly because it was a difficult challenge to create a design for longer materials that would actually get used by teachers in curriculum time. Also more time was put into localising ADAPT materials to partner preferences. However it is clear, from the number of downloads and comments, that these were successful. All Partners ran online courses, with 236 teachers completing: about 65% of the target. However this reflects the challenge of getting teachers to commit to non-essential activities that involve out-of-school commitment.
TRANSFORM outputs
For TRANSFORM, a lot of time was invested in creating a format for a ‘Project’ which teachers would consider of enough value to need several lessons. Two ‘Global Projects’ were created that were published across the consortium, and individual countries created their own ‘Local Projects’ based on locally-relevant issues. The total number of downloads for TRANSFORM Projects was more than 3000. Partners worked in a bespoke yet time-heavy way to individually support 42 teachers with their projects. About 150 teachers were involved in online courses, which is very close to the target. About half of the teachers completed the course.
The Three Phases of ENGAGE
Topicals (ADOPT): 22 of these ‘entry-level’ activities were developed, where students apply science knowledge and enquiry /RRI skills to a socio-scientific dilemma. This is more than the 20 we expected in the DoW. Topicals were very important to our strategy, and publishing them regularly throughout the project has enabled us to build a large user base.
Sequences (ADAPT): 12 of these ‘advanced level’ activities were developed, which explicitly teach students enquiry/RRI skills Sequences using games, and then apply these to solve a socio-scientific Dilemma. In the DoW, we expected to produce 20. The reason for developing fewer activities is that it proved much more complex and time-consuming than anticipated to create a Sequence that teachers would find sufficiently valuable to give 2 lessons of curriculum time for. Therefore we put more effort into creating a smaller number of Sequences to ensure their quality and uptake, which has been high.
Projects (TRANSFORM): A total of 16 extended enquiries into socio-scientific issues were developed, to enable students to work more autonomously using RRI skills. Two of these ‘global projects’ were developed by the WP3 team at SHU, to act as exemplars. These were translated by each partner. Additionally, each partner developed one or two of their own projects: a ‘local project’ customised for an issue relevant to their country. In the DoW we expected to develop up to 20 projects. As with Sequences, we found that development of high quality, project-based learning that would be useful to teachers across all countries was a complex and time-intensive process. Therefore we decided to limit production to the numbers above. Downloads of the global projects have been high and comments very favourable, justifying the emphasis on quality over quantity.
Project Results:
1.0 Deliverable reports
ENGAGE produced a substantial library of deliverable reports, a number of which may have use beyond the immediate consortium members. While a substantial number of these cover consortium specific areas of work such as dissemination and networking plans for the consortium and its outputs, others may be of wider interest. For example Deliverable report D1.1: Responsible Research and Innovation (RRI) curriculum document set out the steps taken to develop the theoretical framework under which the project would be implemented by outlining the current picture of existing RRI curriculum in all participating countries. This aimed to identify barriers and opportunities for the accomplishment of the project’s objectives by carrying out a science curriculum analysis covering 11 European countries, namely: United Kingdom, Greece, Germany, France, Romania, Israel, Spain, Norway, Switzerland, Lithuania and Cyprus for 11-16 years old students. Deliverable reports also set out guidelines and frameworks for the production of project materials for teachers and students, aiming for a common approach that could be adapted for different national contexts particularly around the pedagogy that the project wanted to encourage, teaching strategies and style of student activities and achieving alignment between project objectives and the materials created (see deliverable reports D3.5 and D3.6). The project created an RRI curriculum which aimed explicitly to teach skills like argumentation, which are vital for engaging with socio-scientific issues.
2.0 Website
The project website or Knowledge Hub was at the core of the ENGAGE project. It consisted of a platform in nine languages for 11 countries, which has supported more than 13 000 registered users in carrying out 130 000 downloads of the ENGAGE materials. Each country translated the project website into their own language, as well slides and video about the project, and media repositories of photographs and images, as well as Blog posts. Each country also set up social media channels: YouTube, Facebook and Twitter.
It also provided the platform for delivery online CPD in Massive Online Open Communities (MOOCs - see section 4.1).
The website was constantly updated and improved, in order to keep pace with the project's developments and the consortium's needs. For example:
• sending a notification email to MOOC tutors upon a new user's registration to the course
• supporting MOOC authors to switch to participant's view of the course
• providing a special label for tutors answering discussions in forums
• granting tutors an easy access to the list of participants to the course
• providing a file upload facility for course participants: the Staff Graded Assignment (SGA) module allows tutors to get files from course participants (upon a specific request of assignment) and give them a grade. The grade contributes to the overall course grade as per the specified grading policy.
• supporting the generation of certificates for the courses
It was also adapted to be able to better report on user activity, such as the possibility to download information about all posts at once instead of only per single post; numbers of downloads for a single package, in addition to number of downloads per lesson; data about activity on posts (comments, rates, likes); grouping of downloads per country of downloaders.
A Tag cloud plug-in was set up to enhance the search facility and appeal of the sites. An Enhanced Video Library page allows a quick and direct access from the platform to videos on the national YouTube channel, grouped by Playlists. Videos can be seen directly from the project portal, or reaching YouTube in one click. More than 10 videos about pedagogical tools, materials, RRI experts and Scientists were developed centrally, with some partners also creating their own. X animations were also developed and translated for the websites.
In the second part of the project teachers were able to contribute to the Knowledge Hub by adding details about their activities related to ENGAGE, with users being offered the chance to send their customisations of ENGAGE lessons and have them published on the relevant national site. For security reasons end users could not be allowed to publish posts directly, a form was provided on the website we proposed and discussed with partners two possible policies: providing a form to be completed by users Data are evaluated by the related national site owner and approved lessons are published on the site into a dedicated gallery.
A Partnership Brokering System (PBS) for each national site was set up using the Knowledge Hub as its base. The aim of this system was to support schools to find and select relevant ‘RRI experts’, and created scheduled mentoring partnerships for carrying out the RRI projects which are part of the Transform step of phase 2 deployment. The PBS is implemented as a directory of listings of experts from different areas. The directory is organised in six categories (types of experts): Expert RRI Teachers / Mentors; Media Experts; Science Museum Representatives; Industry Stakeholders; Researchers / Scientists, and Tutors. The main functions of the PBS are: (a) browse the PBS content (lists of experts and categories); (b) view the information of a specific expert or all the experts of the same type; (c) send messages to a specific expert; (d) introduce in the database a new expert and information about him; (e) simple and advanced search for a specific expert.
Learning analytic tools were used for collecting data about students’ opinions, values and knowledge before and after using ENGAGE curriculum resources, for formative and summative evaluation. This was done by embedding quizzes and short surveys within the activities of the curriculum resources. Learning Analytics were used to collect data about teachers’ activities in the ENGAGE Knowledge HUB. Several tools were developed and implemented to collect quantitative and qualitative data about teachers’ behaviour on using resources of ENGAGE project. Data were also collected using external tools (Google Analytics, AWStats), but also by developing and installing supplementary WordPress plugins. These tools provide number of users; numbers of downloads; users statistics by location; new visitors; most downloaded lessons etc. In order to collect qualitative data, quizzes and surveys were developed that were applied to teachers before, during and after different activities (on-line courses, promoting teachers in a new ENGAGE stage etc.).
The ENGAGE Knowledge HUB can be seen as a community of interest where science teachers can learn about RRI from other teachers and experts; they are able to review and explore the materials that other teachers are using and they can propose new strategies for implementation. The reflective comments on the materials and their usage can be found both on the relevant web page, but also in the forum discussions.
Using the data collected by Learning Analytics tools, every partner was able to monitor their local teaching community. The numbers were reported periodically, centralised, and discussed in online and face-to-face meetings. When circumstances required, partners proposed actions and they took measures to guide the local ENGAGE teachers’ community to fulfil the project targets.
3.0 ENGAGE Materials
A central goal of the ENGAGE project was to develop high quality curriculum materials and online course content for the ADOPT, ADAPT and TRANSFORM stages, integrating RRI with the ‘scientific inquiry’ skills that are common to almost all countries’ curricula. The purpose of the three different types of materials was to give a stepped approach to inquiry by introducing teachers to the project principles of RRI with a short ADOPT classroom activity, to then encourage their confidence and progress to the longer ADAPT materials and then on to the longer project based TRANSFORM work with their pupils. ENGAGE prioritised a set of 10 processes which became the learning objectives of our curriculum materials:
• Define questions
• Analyse patterns
• Draw conclusions
• Communicate ideas
• Justify opinions
• Critique claims
• Interrogate sources
• Use ethics
• Estimate risks
• Examine consequences
A clear unique selling point of these materials was the fast track to online publishing of 4-6 weeks, meaning that the materials were very current and the issues involved still fresh in students’ minds when taught in class. This was achieved with a tight development workflow with several stages of tightly scheduled input and consultation from other partners. Localisation of the materials was carried out to meet the needs of each country. Each partner contributed information to make the materials fit their context, and altered the text as appropriate during the translation process. As a result the materials are one of the highlights of the ENGAGE project, in terms of their quality, their dissemination and uptake, and their impact.
ADOPT Topicals - Twenty-two of these ‘entry-level’ activities where students are able to apply science knowledge and enquiry /RRI skills to a socio-scientific issue and then given a stimulating activity for students to review the science and develop an informed opinion. This lesson model was developed hand-in-hand with two Adopt Tools: Dilemma Lessons and Group Discussion (see section XX). Through this integration, teachers using the materials would develop curiosity in how and why the lessons work, and be motivated to enrol in an online course or workshop. Topicals were very important to the ENGAGE strategy, and regular publishing of these throughout the project has enabled the building of large user base. As lessons these would last for one 40-60 minute session.
Examples:
Text neck - New research suggests that smart phone use is seriously damaging our necks. Should use smart phone less to prevent neck damage?
Eat Insects - Farming large animals uses precious resources. Can you persuade people to swap meat for insects?
Invasions - Common ragweed is an invasive plant which is spreading across Europe. Should we control it by introducing non-native beetles?
ADAPT Sequences - Twelve of these ‘advanced level’ activities were developed to explicitly teach students enquiry/RRI skills Sequences using games, which can then be applied to solve a socio-scientific Dilemma. These lasted from 1-2 lessons.
Examples:
Electronic cigarettes - Campaigners in support of ban point out that nicotine from e-cigarettes may contribute to heart disease and cancers, as well as damaging the brains of developing foetuses. In this activity students decide whether they support a ban. They apply their knowledge of particle theory to decide whether exhaled nicotine can reach non-vapers nearby, and then learn to judge risks to decide whether the benefits of a ban on indoor vaping outweigh the risks.
Two degrees - UK winters are getting wetter and flooding is a common threat in many parts of the country. Scientists believe climate change may have caused this extreme weather. In this sequence students apply their knowledge to create an apocalyptic weather report. Then they learn the skill of examining consequences, and judge solutions for limiting the temperature rise to 2 degrees.
TRANSFORM Projects - A total of 16 extended enquiries into socio-scientific issues were developed, to enable students to work more autonomously using RRI skills. These were translated by each partner. Additionally, each partner developed one or two of their own projects: a ‘local project’ customised for an issue relevant to their country. Downloads of the global projects have been high and comments very favourable, justifying the emphasis on quality over quantity. These projects were designed to take place over the course of several lessons. Transform activities allowed partners to share valuable experience in innovative teaching practices. A gallery of “Transform-like local projects” on the Knowledge Hub illustrated a very high degree of creativity across the consortium, and the benefits of sharing practices.
Examples:
Exterminate: Mosquitoes are the world’s most dangerous killer. The diseases they transmit; malaria, Zika and dengue fever, cause more than a million deaths per year. Some scientists have suggested exterminate all dangerous mosquito species. One method is to release genetically modified (GM) male mosquitoes which prevent further breeding. In this project, students investigate whether exterminating mosquitoes is a good idea, using scientific knowledge about interdependence.
Eco-phone - As the number of smartphone users worldwide exceeds 2 billion, and as users update their devices with ever-increasing frequency, there are growing concerns about the impacts of smartphone manufacture and disposal on the environment and human health. In this activity, students use knowledge about Earth resources, as well as applying their working scientifically skills, to work out how to persuade phone manufacturers to make eco-friendly smart phones from sustainable materials.
The feedback received about the materials has indicated they act as helpful exemplars for teachers unfamiliar with RRI teaching. However, in some countries where there is not a culture of using pre-prepared lessons, such as Greece and Cyprus, some teachers were less keen and want either to customise them, or improvise entirely new activities. This was encouraged by delivering materials in a format that allowed teachers to make their own changes and via teacher guides, workshops and courses to clearly state the project rationale, so teachers could make changes while keeping pedagogical principles intact.
3.1 Areas of commonality and localisation in Materials
The materials were centrally designed in an effort to have a common base between the materials used in all the partner countries. Since the main purpose was to design materials that promote RRI, all materials were designed based on a common framework that had been agreed within the consortium on what is defined as RRI, and the different types of skills that are considered important in the teaching of science. More specifically, the following aspects of the materials are common:
● Emerging science and technology contexts
● RRI goals
● Structure and design of materials
● Core Science Ideas
Despite the decision to have a common structure and design for the Materials, it was considered important to localise the Materials to meet the individual needs of each country and their science curricula. The main reasons for localisation:
● An initial review of the curriculum of the partner countries demonstrated different content and emphasis on different skills.
● Research has shown that when the materials are relevant to students’ lives, science is more engaging. Therefore the effort during localisation is to use information and data from each partner country to make the materials relevant.
● RRI includes ethical and moral aspects, and these aspects are different in each country.
Aspects of materials which are localised:
● Specific contextual information (e.g. information about the sources of energy used in each country)
● Scientific data (e.g. data about consumption of energy in each country)
● Science curricula links (each partner provides the links to the national curriculum)
● Additional content (e.g. according to the national curriculum there might be links to further knowledge)
● Web links for news stories and media (e.g. news stories from the local press from each country)
3.2 Beyond the ENGAGE consortium
Use of ENGAGE materials has spread to 80 countries, based on data of downloads from the website. In Vietnam, China, Peru, Columbia and Uruguay, we have received direct reports of schools taking up ENGAGE and embedding into their curriculum. The largest impact has been in Brazil, where 4 different cities ran very large projects with schools, using the ‘Zika’ ENGAGE material which is so relevant in Brazil. Around 80 teachers and more than 1000 students participated in workshops, and students presented posters at a national exhibition. Equally important, the students collaborated with 36 scientists and experts to develop their understanding of RRI.
4.0 Continuing Professional Development (CPD)
ENGAGE used a combination of face to face workshops and on-line course and webinars in order to deliver effective Continuing Professional Development to teachers. It was evident that a rigorous professional development framework was needed to inform the development of the CPD workshops covering guiding principles, strategies, prototypes and pedagogical design. The framework describes each element of our PD programme, and provides a clear objective and rationale for the strategies chosen for ENGAGE. Second, for teachers who were curious to find out why the lessons worked, we developed workshops and online courses to teach the tools for using the materials effectively
Important outcomes of the workshops included;
• familiarisation of teachers with dilemma and small group discussion RRI teaching strategies;
• hands-on engagement with the Adopt materials;
• motivation to participate and engage in the on-line courses for more in-depth understanding and knowledge co-sharing.
The starting point was to describe what ‘practices’ were needed from ENGAGE teachers at the end of the project;
• Use authentic tasks to help students apply science learning to every-day life
• Explicitly teach the ‘RRI skills and knowledge’ needed to deal with science issues
• Use open dialogue to build students’ reasoning and understanding
The latter of these is probably the hardest practice to implement involving a big shift in interaction style from one where the teacher’s views and ‘correct answers’ are dominant, to one where the students’ ideas and argument are more important.
CPD tends to be more effective when it focuses on well-defined skills rather than general practices; therefore ENGAGE followed a similar approach to defining the skills as used in the ‘Ambitious Science Teaching’ project and the FP7 inquiry project ‘TEMI’. This aims to turn the practices into a small number of easy to use ‘Tools’ for teaching. Six tools were developed - Dilemma Lessons, Problem-solving Sequences, Scenario-based Topics, Group Discussions, Class Conversations, and Performance Assessment. Instructing and coaching teachers in the use of the Tools in the classroom was the focus of the CPD ENGAGE programme.
Two CPD workshops were conducted in each of the 11 countries of the consortium, on the dilemma and group discussion teaching strategies, engaging a total of 1268 teachers. In addition partners also rolled out dozens of workshops that engaged more than 1200 teachers.
In addition in Cyprus, Romania, Greece and the Netherlands, partners have managed to incorporate ENGAGE into both undergraduate and post-graduate initial teacher training, and Masters courses for teaching and science communication. This includes using specific ENGAGE lessons and ‘Tools’ for RRI-based teaching within the course programme, so that many new teachers will be exposed to ENGAGE at the beginning of their careers, and are likely to integrate our approaches into their repertoire.
4.1 Massive Online Open Communities (MOOCs)
Based on the professional development framework above and in addition to the workshops, online courses were created for each project stage (ADOPT, ADAPT and TRANSFORM). Twenty five of these courses were delivered across 11 countries, with 974 teachers enrolled in total.
The online courses in the partner countries were based on common content but differed between contexts in terms of the structure, the mode of delivery and the facilitation. This locally relevant approach to delivery was decided upon in order to make courses most relevant to different national contexts and teachers attitudes. In the UK for example methods used for the delivery included forum posting, review questions, assignment activity and webinars. The Greek and the Spanish online courses, on the other hand, followed a more structured and rather traditional mode of delivery, with content, tasks and assignments being uploaded each week by the course facilitators. In the Spanish courses, each teacher created their padlet (https://padlet.com/) to upload lesson plans with ENGAGE materials or RRI CPD tools, as well as photos from the lesson, student products and lesson reflections. Each teacher had to comment on the padlet from another teacher as part of the activities to complete the course. Facilitators commented on the padlets as well. The above are examples of how the courses in different cases supported interaction with other teachers and facilitators.
The success of these courses varied across the consortium. In some contexts the courses were very successful in getting teachers to use the materials, and reflect on the pedagogies of teaching with RRI. However, in others, it proved difficult to recruit many teachers and have them complete the course. Around two thirds of teachers participating reported that they learned strategies for developing students’ inquiry skills to a high extent, and half the teachers said they learned how to structure effective discussions.
5.0 RRI festivals
Eleven countries organised a final ENGAGE event, allowing teachers to meet, present results of local Transform projects, discuss open schooling and innovative pedagogies, as well as RRI topics related to science education.
Andy - need more on these???
6.0 Internal evaluation
The internal evaluation of ENGAGE focussed on the impact of the project's CPD activities and innovative teaching materials on teachers as the core target group, while also carrying out a more limited impact assessment for students given the drive to improve their knowledge and skills via teacher involvement. The evaluation included both quantitative and qualitative data and was driven from within the consortium.
A pre and post ENGAGE CPD questionnaire was used to examine 56 teachers' responses (from five countries - Romania, Cyprus, Lithuania, UK, and Spain ) against the five dimensions of RRI teaching established by the project:
• Teachers’ Knowledge of RRI
• Teachers’ conception of their own role
• Teachers’ use of discourse
• Teachers’ conceptions of learning goals
• The nature of classroom activities
This established that there was a positive shift in all five dimensions in the sample of teachers. In particular there was only a statistically significant positive shift seen in Dimension 1 "Knowledge of RRI".
In terms of qualitative research with teachers a descriptive qualitative study was designed to investigate the change in teachers’ Pedagogical Content Knowledge. This used three instruments - a Lesson preparation form (assessing PCK before use of ENGAGE material/workshop participation), Lesson reflection form (assessing PCK after use of ENGAGE material/workshop participation) and an Observation table to support PCK data collection, which were refined after a pilot study was carried out.
Thirty one teachers from four partner countries (Spain, Norway, Israel and Cyprus) participated in the PCK study. The results are summarised in Table 1 below.
Table 1. Summary Results of teachers’ PCK using ENGAGE materials in five countries
Country (N-teachers)
Goals in line with ENGAGE
objectives
Focused mainly on skills
Focused mainly on science
Aware of
specific student difficulties
Adapted the material to fit their students
Prepared assessment activity /
tool
Changed PCK (main
aspects*)
Spain (7)
5 teachers
1
1
3 (misc., beliefs)
3
1
7 (IS, US)
Norway (7)
3 teachers
2
2
6 (skills, beliefs)
1
0
3 (IS)
Israel (10)
4 teachers
6
0
5 (skills, beliefs)
2
0
10 (IS)
Cyprus (7)
6 teachers
1
0
6 (misc., beliefs)
6
0
4 (IS, US)
*IS =Instructional strategies, US= students’ understanding
As illustrated, 18 teachers articulated goals that were in line with ENGAGE objectives (i.e. students apply science to new contexts, use higher order thinking to form evidence-based opinions on societal needs and values). It is interesting to note that the analyses showed that years of general teaching experience had no direct correlation with the quality of the teachers’ PCK. The main conclusion is that by using these ENGAGE materials some of the teachers changed aspects of their PCK related to ‘Instructional Strategies’, and ‘students' understanding of science’. Analysis of individual teachers’ profiles suggests that these changes are linked to teachers reflecting on specific difficulties they faced during teaching - in particular difficulties linked to coordinating argumentation activities, group work and discussing uncertainty. Moreover, in all countries, the less-developed PCK component was about ‘Assessment’. This might be linked to the fact that most of the teachers implemented the ENGAGE lesson as a separate lesson, not linked to their curriculum. This may suggest that PCK can shift by using educative materials, without necessarily engaging in long-term professional development.
The way of teaching science
We conducted an evaluation survey of users of ENGAGE materials, across all partner countries. 545 teachers responded. The majority (44%) of the teachers reported that the ENGAGE materials enriched their way of science teaching ‘above the average’, and the second biggest (30%) group of the teachers reported this as to a ‘great extent’.
In-depth case studies
Each partner carried out one or more case studies to focus on teachers who had used ENGAGE. Overall, from the case studies, it was found that ENGAGE materials enabled teachers to think creatively about how to teach science. Some teachers also learned to teach by starting with a good question, making the students think by themselves and get them to find a solution to the Dilemma.
Online Course (MOOC) impact
A total of 297 teachers took part in the evaluation of the online courses, the results of which are given in Table 2.
Table 2 - results of MOOC evaluation
Teacher learning/skill development
% agreeing
learned strategies for developing their students’ inquiry skills through ENGAGE online courses at a high extent
55%
mastering the use of 5E lessons to develop RRI/enquiry skills
52%
learned how contemporary science can engage students and get them thinking and talking through ENGAGE online courses at a high extent
53%
understood how to use ‘student’ thinking guides to scaffold RRI/problem-solving skills
51%
learned how to prepare students for effective discussion by building an argument based on evidence and reasoning
51%
effective discussion using different components of argumentation and designing their own problem-based lessons with argumentative conversation for RRI/enquiry skills
49%
learned to assess when inquiry or working scientifically could be used within the curriculum
44%
Potential Impact:
Please provide a description of the potential impact (including the socio-economic impact and the wider societal implications of the project so far) and the main dissemination activities and the exploitation of results. The length of this part cannot exceed 10 pages.
The Impact and legacy of ENGAGE
Lots of ENGAGE users in some countries
The extent of take-up of ENGAGE materials has been substantial. In Lithuania, 16% of all science teachers are estimated to have used ENGAGE. In the UK, a third of all science teachers have registered and more than 10% have so far used the materials, based on survey data. These numbers indicate that there are a good number of teachers who, because of their positive experiences, are likely to spread ENGAGE by word of mouth so that the number of users continues to rise.
Effective online courses and workshops
The ENGAGE online courses were a localised success in countries such as Spain, Greece, Israel and Cyprus, getting teachers to use the materials, and to reflect on the pedagogies of teaching with RRI. However, in other countries it proved difficult to replicate the success, often falling down at the recruitment of teacher stage. Around two-thirds of teachers participating reported that they learned strategies for developing students’ inquiry skills to a high extent, and half the teachers said they learned how to structure effective discussions. In Greece, in-depth evaluation found a clear positive impact on raising teachers’ awareness of RRI in science education. In some countries, workshops were an easier way to get teachers to participate in formal professional development. The internal evaluation found these were also influential in helping teachers gain confidence in using group discussions and dilemmas in science lessons.
Evidence of a shift towards RRI-based teaching
The internal evaluation, carried out by Tu Delft, found evidence that teachers shifted their practice towards RRI-based teaching as a result of involvement in ENGAGE. Using a 5-dimensional model to characterise this shift, and from the statistical analysis, there was strong evidence that ENGAGE could improve teachers’ knowledge of RRI. In the other dimensions, such as teachers’ roles and their goals for science teaching, there were only slight increases in the mean values. With limited time to implement the TRANSFORM stage most of the evidence was captured from ADAPT teachers. They had modest experience of ENGAGE compared to the TRANSFORM teachers who had been through the whole programme. A more significant shift in RRI-based teaching may have been found if more TRANSFORM teachers had been surveyed.
In the research into teachers’ pedagogical content knowledge (PCK), it was found that using ENGAGE materials enabled some teachers to successfully develop new instructional strategies for addressing socio-scientific issues. The internal evaluator concluded that ENGAGE’s materials were sufficiently educative for teachers to learn new strategies for RRI-based teaching without engaging in formal professional development.
Increased student motivation and inquiry skills
The thousands of comments on the website provide anecdotal evidence ENGAGE makes science lessons highly motivating for students. Our external evaluator found some evidence of a positive influence on student ENGAGEment from video recordings of two teachers’ lessons. When students were experiencing an ENGAGE lesson, their ‘time on task’ seemed to improve compared to before the lesson. In Romania, the impact of ENGAGE on students’ behaviour was studied with 64 students. Anecdotal evidence suggested that after using ENGAGE the majority of students began to read and discuss things related to science and technology more outside of classroom. Each partner also carried out case studies on their teachers’ use of ENGAGE, and in all cases using ENGAGE seemed to improve students’ inquiry skills. However, in terms of other stakeholders such as scientists, the evaluation was unable to show impact.
Continuing usage of ENGAGE materials
The ENGAGE website and its collection of curriculum materials is a major legacy for the project. It will be maintained for three years, with 37 high quality curriculum materials in 11 languages to help teachers use socio-scientific issues with their students to develop RRI skills. The materials are backed up by articles and discussion papers for teachers to deepen their knowledge of RRI. In most partner countries, the online course content will continue to be accessible for teachers to participate in a localised way. Based on the 16,000 teachers already registered, and word of mouth, it is hoped that the numbers of downloads, usage and comments will continue, even without partners’ direct involvement.
Impact on science curriculum policy
ENGAGE was particularly successful in influencing curriculum policy in the UK, working with the largest awarding body (AQA) to incorporate ENGAGE’s ‘RRI curriculum’ into its Science Syllabus for 11-14 year olds. The Syllabus promotes ENGAGE activities as ideal resources towards meeting one of the assessment objectives - to apply knowledge. AQA and other publishers such as Hodder have specifically recommended ENGAGE on their websites, and this will ensure that many teachers use the materials as a core part of the curriculum for several years.
Embedding into teacher training
In Cyprus, Romania, Greece Lithuania, UK, and the Netherlands, partners have managed to incorporate ENGAGE into both undergraduate and post-graduate initial teacher training, and Masters courses for teaching and science communication. This includes using specific ENGAGE lessons and ‘Tools’ for RRI-based teaching within the course programme, so that many new teachers will be exposed to ENGAGE at the beginning of their careers and are likely to integrate the approaches into their teaching.
ENGAGE beyond consortium countries
Based on data of downloads from the website, the use of ENGAGE materials has already spread to 80 countries. In Vietnam, China, Peru, Columbia and Uruguay, there are direct reports of schools taking up ENGAGE and embedding into their curriculum. The largest impact has been in Brazil, where four different cities ran very large projects with schools, using the ‘Zika’ ENGAGE materials which are of obvious relevance in Brazil. Around 80 teachers and more than 1000 students participated in workshops, and students presented posters at a national exhibition. Equally importantly, the students collaborated with 36 scientists and experts to develop their understanding of RRI.
ENGAGE further potential Impact
Below are a series of promising examples of different aspects of ENGAGE that offer future potential in terms of offering potential for localised legacy. We would rate these as all tentative yet positive aspects with future value to them. Including
Website – Romania, France, Germany, Lithuania, Switzerland, Israel, Norway, Greece, Cyprus, Spain and the UK.
Changed teacher Pedagogy – The Romanian experience
A key achievement of ENGAGE is the high level of Engagement of the Romania partner with government officials at different levels of the education infrastructure which ensures they play a major role in discussions about science curricula and teacher training.
As stated in the DOW, the project expected at least 2 pre-service institutions and 2 in-service training providers to have integrated elements of the ENGAGE programme in their pre/in-service training, within each country.
Partners have worked with teacher training institutions to embed exposure to and use of RRI pedagogy through ENGAGE OER and guidelines for teaching. The degree of impact has varied across partners. The evidence suggests that a number of factors account for this, but three key factors are the expertise of the partners, the institutions they are located in, and their role within these institutions. This configuration of factors was not evident with all partners.
A prime example of achievement that is being further developed is Romania. The partner works within a university that provides teacher training. A key role its three members have played for several years is in the development and delivery of pre and in-service teacher training. The Romanian partner achieved wider penetration across education levels, disciplines, and across the country than may have been expected.
First, the partner was able to integrate ENGAGE materials and RRI techniques into the curriculum at undergraduate and postgraduate levels and provide a number of sessions for pre and in-service teachers. Second, the ENGAGE project was focused on secondary education but the partner was able to integrate RRI pedagogy and some elements of the OERs into teacher training for teachers of pre-school and primary levels. Third, ENGAGE focused on science education. The partner was able to include RRI pedagogy and some elements of the OERs into a range of disciplines such as Maths and other programmes in the Faculty of Orthodox Theology as well as in programmes in Geography, Food Processing, Economics, Technological Education, Education for Health, Counseling, Career Orientation, Religion and Music.
Finally, there is potential for wide geographical reach as there is a very high probability that this new approach will influence the development of teacher training across Romania. Discussions with Inspectors of Science subjects at county level, and with government officials working in national teacher training development programmes and national education policy levels, testifies to this. The partner and other colleagues at the university have a strong pre-existing relationship with policy officials and professional teacher associations. The views of the partner are respected and they were involved in frequent bilateral and multi-lateral meetings concerning teacher education. As teacher education is very tightly controlled in Romania with teachers having to undertake nationally accredited training to both obtain and maintain their teacher status, the fact that the partner has a seat at the table, both county and national, and is able to draw on its experience of delivering ENGAGE, is an excellent achievement for the project.
The School Science Curriculum : The UK experience
The AQA KS3 science curriculum has been developed in conjunction with the enquiry aspects of Enquire, Analyse, Communicate and Solve all have a strong link to the ENGAGE model.
The UK partner has had a significant and major impact on the science education curriculum in the UK. It has ensured that the ENGAGE framework for science teaching has been adopted by one of the UK national exam boards, AQA. While not the only exam board in UK, the AQA is a major board. Both the mobilisation of existing contacts and serendipity played a role in making this happen.
The new Key Stage 3 (for 14-16 year olds) Science Syllabus from AQA in the UK shifts the focus of science education from the acquisition of knowledge. Jointly developed by ENGAGE partners, its breakthrough design reduces the science knowledge content teachers need to cover, and shows them how to put science in context by giving students the skills to apply their knowledge to real life. It is hoped that this will help make learning science an exciting experience which will inspire current and future generations of students.
The UK partner has been able to use the AQA branding on its website and to publicise and run its events. At the same time, the AQA website provides additional credibility by links on its official website to the ENGAGE Knowledge Hub.
At the RRI festival that the UK partner had in March 2017, the Head of the AQA Science section gave the keynote speech. This link between curriculum and ENGAGE is a powerful one that will give momentum to ENGAGE resource uptake in the UK going forwards.
Website and knowledge hub – All partner and beyond experience
Work Package 2 lead by the SME Lattanzio learning partner No based in Genoa Italy. The Lattanzio learning team were well supported in their website realisations by the team at SHU and OU. Our materials are accessible and available through 11 different languages. In 2017 ENGAGE won a worldwide Open Education award.
A major achievement of the project is the Knowledge Hub, located on an attractive, user-friendly website. This houses an extensive range of Open Educational Resources (OERs) in English and the nine other languages of partner countries. One OER has been translated into Portuguese and another into Arabic. It has won international recognition through a prestigious award. The Knowledge Hub has around 18,000 registered members, but well over half are from UK. There have been approximately 130,000 downloads, and although all the partner countries have had their resource materials downloaded, the majority are downloaded from the English language webpage. Teachers from 85 countries have accessed the website demonstrating the wide reach of the project. There has been a high degree of exploitation of the OERs, which have been widely disseminated though digital and print channels, and the profile of the project has been raised through conference presentations and publications. While there is no robust evidence collected by the project to confirm that the target of 11,750 teachers using the materials has been reached, a guesstimate based on the number of registered users and downloads suggests that the target may well have been met. The website will be maintained until 2020 and there is a strong likelihood that the web traffic will continue to rise. The functionality of the website is high and offers potential as its online presence will be supported for the following 3 years. The website should act as a driver for future CPD activity and further teacher Engagement in some countries will continue to grow through word of mouth by teacher users recommending it to other teachers.
The Lattanzio group (ELS, Italy) provided Knowledge developed the Knowledge Hub (KH), namely the technical platform supporting the project’s main activities. The work focused on the development of the project website, the materials platform and online community. It also included the development of the virtual environment for the online courses and the learning analytic tools to record the use of materials and online community interactions. Complementing the work undertaken by the Lattanzio group was VUT (Romania) and the OU were also involved in the development of the EdX platform and analysis of technical aspects in order to assure a secure and trusted on-line environment for the KH.
It is commendable that the Lattanzio group paid considerable attention to designing the website and continuous improvements have been made to it throughout the project to increase its functionality and user friendliness. As they pointed out:
From our side, we tried to be as supportive as possible. When partners asked for technical assistance, we provided user guides and video tutorials, showing the procedures step by step and offering to assist live in Skype via screen sharing. (ELS Italy)
The KH has been enriched by new technical functionalities which have promoted more user-generated content as follows: (a) the rating system allows new users to identify appropriate materials more easily (b) teacher ability to respond to other comments has facilitated the development of reflective conversations (c) the personalization function reminds users to make comments on the last downloaded materials and the comments made by an expert teacher can be highlighted by adding the ‘Expert’ tag and (d) the brokering system contains the list of all experts, presents the expert profile and provides a way to establish contact between users and a specific expert. Partners could monitor progress and acquire analytical data on a regular basis which enabled them to make changes to improve accessibility to materials and promote better interaction with users.
Despite the availability of the improved functionality, the full potential has not yet been realised by all partners and users.
Use of the Knowledge Hub by Teachers outside the Partner Countries
The availability of OERs in ten languages increases its accessibility to beyond the English-speaking world. Most web traffic is evident on the English language page which reflects the importance of English as an international language and medium of instruction in many countries. The users of the website come from around 85 countries including Nigeria, Indonesia, China, India and Brazil. Some of the teachers from these countries who have accessed the website are local people, while many others are foreigners with English names which suggest they may be British teachers working overseas. Some of the teachers have downloaded the materials but others have merely registered as users.
One of the intended legacies of the project is use of its resources by teachers worldwide. While the geographical reach achieved is in itself a notable achievement, greater impact could have been achieved by engaging with these foreign and local teachers to help them use the material and also to mobilise them to act as country hubs. An attempt in this direction has been made. The most commendable, and likely to have sustained impact, is the work that ENGAGE has done with Brazil. The Zika resource material as with others such as 2 degrees show the value of rich context based contemporary resource materials to generate purposeful teacher and student discussion whilst developing RRI skills.
A key learning outcome for the co-ordinating partner (SHU) and project management coordinator (Andy Bullough) has been that centrally developed resources can work really well across delivery partners in different countries as long as they offer the dissemination partners the opportunity to localize and personalize the materials to fit within their own national contexts. This international project is the result of the laudable mobilisation of the contacts in Brazil of an ENGAGE team member working at the Open University.
The conceptualisation of this project and its scale went way beyond anything attempted with the ENGAGE project. The project ran for a month and used 36 Brazilian research educators as well as teachers and scientists. There were 1,473 learners including deaf people, older citizens, secondary and primary schools and members of local communities.
The project with Brazil investigated how ENGAGE materials could be easily disseminated to promote inquiry skills for RRI in different areas of Brazil and promote collaborations between universities and schools. Participants used the ENGAGE ‘GM decisions’ and ‘risk’ games to develop informed based opinions about genetic modified food through ‘Open schooling’. This means schools working in cooperation with researchers, parents and experts, and becoming an agent for community well-being. Families were encouraged to become real collaborators by interacting through social media and events. Biotechnologists and agro biodiversity consultants working on GM and wider society were also involved in discussing real-life projects in the classroom.
The Brazilian educators found that the ‘GM decisions’ and the ‘risk’ game could be easily embedded in the Brazilian curriculum but it required proper planning. Students interacted with researchers and scientists and co-created various examples to communicate their results. The project’s achievements included the creation of various products: 1 exhibition, 9 games, 4 new OERs, 42 illustrations, 1 webinar, 28 concept-maps, 1 sign-language activity for deaf people, 2 posters, 2 workshops and 4 video clips. The project concluded that inquiry skills for RRI can be fostered in many Brazilian states.
The work with Brazil has been a very good outcome for the ENGAGE project. It is likely that many of the OERs will be translated into Brazilian Portuguese for use by teachers all over Brazil. Indeed, the OER called Exterminate has already been translated and sits on the English webpage, as a webpage in Portuguese does not exist. The OER is very relevant to Brazil as it focuses on mosquitoes and many of the diseases they transmit, such as Zika, are rife in Brazil. The teachers and other stakeholders involved in the Brazil project have the potential to form valuable and sustainable hubs supporting communities of practice.
Another small but laudable attempt to make ENGAGE materials available to speakers of other than the ten languages on the KH is the case of Israel. The webpage of Israel, which is in Hebrew, also includes one OER, Ban Cola, translated into Arabic. This will be very valuable for Arabic-speaking teachers who have attended events organised by the Israel partner but have not been able to follow up with their Arabic-speaking students because they did not have resources in the language. It would be very helpful if funds could be obtained to enable all the materials to be translated into Arabic. This would be of benefit not only to the Arabic speakers in Israel and Palestine, but also for the hundreds of Arabic-speaking migrants who live in the partner countries.
The statistics on the website on the large number of registered users indicates a community of interest. Even if they do not download the materials or leave comments, a large number of teachers and science educators from the partner countries, supplemented by a few from a wide range of around 85 countries, have been exposed to the project and to RRI teaching. Some teachers who have not used the materials may, nevertheless, have used the RRI pedagogic approach and applied it where they could to their existing teaching practice. For instance, they may have involved their students through using the idea of a dilemma. Hence, the impact on some teachers has been of the RRI pedagogic approach, whereas for other teachers, they have not only adopted the pedagogic approach but also used the OER materials.
The Lattanzio group will support the maintenance of the website until 2020. It is anticipated that the web traffic will increase over time. Some indication of this is evident from the fact that in February 2017, 15,000 users were registered on the KH but towards the end of March 2017 this had risen to 18,000 – an increase of 3,000 in about a month. There are precedents for this pattern of increased traffic. Two examples are given here.
A project in which the ENGAGE PD was called science upd8 – a forerunner to ENGAGE - had continued web traffic of about 5,000 visitors per month for 5 years and there were downloads of the materials after the development finished. The ENGAGE project co-ordinator was involved in 2007-2010 in a project called cre8ate maths. According to him, the cre8ate maths website had approximately 2,000 users at the project end in 2010. This number increased through word of mouth to nearly 6,000 users (signed up members downloading content) by 2012m and as recently as 2015, still had 4,000 active users. Since the end of the project in 2010 no updating of the website has taken place.
It will be interesting to see if a similar trend will be manifested on the ENGAGE KH. There are some steps being taken to stimulate the growth of users. For instance, to increase the impact on countries worldwide, the UK partner is planning to work with the British Council in UK and with their multiple local offices worldwide, to increase the publicity and use of the KH. Another step, which all partners have undertaken, has been to disseminate their individual language based resources widely to a range of online libraries and repositories. Furthermore, each partner can load the resources onto the website of their own institutions. There have also raised the profile of ENGAGE through conference presentations and publications and many partners expect to continue with these activities. These wide dissemination activities, while noteworthy in their own right, will also draw teachers to the KH where they can make contact with other users, read and leave comments, gain ideas about how to adapt the lessons for their own students and access experts.
If the potential of the KH is realised over the next few years, then with its high quality open education resources and multiple functionalities, it is is likely to be one of the most enduring and significant achievements of the ENGAGE project.
List of Websites:
http://www.engagingscience.eu/en
