Final Report Summary - GENDERC (Gendered dimensions in ERC grant selection – gendERC)
Executive Summary:
To explain lower success rates of female applicants in ERC grants, we collected data about past performance of the applicants and interviewed panel members about how selection criteria are practiced in general and specifically for female vs. male applicants. Controlling for past performance, we found gender bias. The analysis of the interviews provides empirical evidence that current evaluation practices are suboptimal, leading to some gender-biased (gendered) practices. Providing empirical evidence of gendered practices raises gender awareness and gives a chance to modify and improve them.
When analysing potential sources for gender bias in the ERC peer review process, we found that gender bias exists (after controlling for the PI’s past performance) and that it can be located on two different levels. Evidence-based recommendations for improvements have been developed on both levels.
As we identified some general suboptimal evaluation practices – due to a vague definition of excellence and a quite open process how to apply different elements of excellence in practice – it is recommended to (1) more specifically define what excellence means in the context of ERC as a prestigious funder for blue sky / frontier research (e.g. at the panel level). To (2) better standardise the process will reduce space for individual interpretations and improve the selection process.
Such interventions would also help to reduce bias on a second level: gendered practices – which affect female and male applicants differently – have been identified. Treating men and women differently is mostly done unconsciously and related to personal assumptions and attributions on how men and women are or should be (=gender stereotypes). When in some panels independence or mobility are checked for women only while they are not questioned for males – for the latter, they seem naturally given – male applicants are favoured, different standards are applied. To change this pattern, room for reflections is needed to (3) raise awareness for gendered practices. Therefore, unconscious gender bias checks or trainings, films and more can be provided and offered to panel members, ERCEA staff and also the ERC as a funding institution in an efficient format. By (4) recognising that gender plays a role in the peer review process and when constructing excellence, ERC could take over a pioneer role for more gender equality in science and for better selecting the best (female and male) applicants.
We hope that our evidence-based arguments contribute to better understand what is going on in the ERC peer-review process and in panel meetings and to illustrate where and how the quality of the peer-review process in general and gender fairness in the evaluation process can be optimised.
Project Context and Objectives:
In this project, we investigate the extent and sources of gender bias in grant reviewing schemes, using ERC Starting Grants and Advanced Grants as case studies. The project focuses on the three main aspects of the schemes: the formal requirements and procedures, the composition of the reviewers and panels, and the process of application and evaluation. More specifically the following questions are answered: (i) Does the application process lead to gendered self-selection? (ii) Where in the selection process are (potentially) gender-biased decisions made? (iii) Are the decision criteria gender-biased, and/or are they deployed in a gender-biased manner? (iv) To what extent is this due to the (gender or topical biased) composition of the panels and/or reviewers?
This project has two general objectives:
(1) To identify potential (in-)direct discrimination in institutional processes and practices of ERC funding activities due to the gender dimension
(2) To analyse the causes of gender bias in grant allocation, in order to support gender-neutral selection processes at the ERC level, and as an example for other funding agencies
More specific objectives are targeted in the different work packages:
(a) To identify potential biases in applying for ERC Starting Grants and Advanced Grants
(b) To clarify the dimensions of the concept of excellence as it is de facto used in peer review and panel review (WP2, WP3)
(c) To identify the gendered elements in the conception of excellence, that is those elements that work negatively for women (WP2, WP3, WP5)
(d) To identify gendered practices of deploying the excellence dimensions (WP5)
(e) To determine the size of gender bias in ERC grant decisions, and contributing to a reliable and valid method for measuring gender bias. (WP6, WP7)
(f) To contribute to a gender-neutral excellence concept and to procedures of a gender neutral way of deploying the excellence criteria in practice (WP5, WP7)
(g) To identify the phases in the review process where gender bias increases and/or decreases (WP6)
(h) To clarify the process of peer reviewer and panellist selection, and to identify the gendered and topical bias in the group of reviewers and the group of panellists (WP4)
(i) To understand the causes of gender bias, and especially the role of
• gendered criteria (WP3)
• gendered application of neutral criteria (WP5, WP4)
• gender-biased panels (WP4, WP5, WP6)
• topical-biased panels (WP6)
(j) Proposing a peer and panel review procedure that is not gender-biased (WP7)
Project Results:
1 Context: Success rates in ERC panels
Starting point for our study was the fact that in all ERC grants female applicants have lower success rates (relation applicants/grantees) than men, especially in the LS domain. We have looked at all nine LS panels of the StG 2014 and analysed the success rates in step 1 and step 2 of the evaluation process – since differences can be located more accurately and potential improvements can be installed more effectively. In this case, women have a 6% lower success rate in step 1 and 2% in step 2, which makes an overall negative difference of 3.
2 Methods and data
Apart from controlling past performance we used a mix of quantitative and qualitative data to provide insight in the nature and prevalence of the bias. Our findings are based mainly on qualitative interviews conducted with 32 members of ERC peer review panels. Additionally, we analysed ERC policy documents and sent an online survey to all applicants who had given informed consent to be contacted. More than 1.200 answered the questionnaire. Further sources of data were evaluation reports from panel members and external evaluators, CVs of applicants, and Web-of-Science data for about 40% of the applicants and for panel members of four panels. We used the data to test our model, in which success is explained by performance (quality) of the applicant and gender.
In the summary below, we refer to the deliverable when mentioning our quantitative analyses. When citing interview data or other literature, this becomes clear through quotes and references.
3 What is gender bias?
Gender differences are not the same as gender bias. In order to preclude the possibility that different success rates are due to differences in past performance, we collected the performance data for all applicants of 4 panels, and about 30% of the applicants of the other 5 panels. Of the four panels, three had similar patterns and the fourth was rather different. When controlling for past performance, gender remained having a significant effect on success, in favour of men. In the fourth panel, it was exactly the other way around. At the LS-domain level, the gender bias (in favour of men) is still visible, but non-significant. For PE and SH panels we only have the results at domain level, SH is similar to LS. The following conclusions can be drawn:
- Gender bias exists.
- It exists in two directions, but more often in favour of men.
- At field/domain level, the opposite effects partly even out; the remainder effect still is in favour of men (LS and SH), but less strong.
- In PE, gender effect at the domain level is about absent.
Now, how can this gender bias in the evaluation procedure be explained? For that we did some quantitative analyses and heavily drew on the analysis of the interviews with panel members. Finally, we again emphasise that observations are essential.
4 Excellence – gendered indicators
As ‘Excellence’ is claimed to be the core criterion in the evaluation process of ERC frontier research grants (see Work programme 2014: p. 20), the understanding and operationalisation of excellence were a major analytical step in our study.
So far, mostly quantitative metrics function as criteria for excellence to demonstrate an applicant’s scientific performance and success. Publications and impact points have become of increasing relevance and reflect the meritocratic understanding that quality of science can be measured by transparent and fair indicators. Yet, it has been widely demonstrated in recent research that the concept of excellence and the criteria for excellence are socially constructed and not gender neutral. Male-dominated networks are relevant in recruitment processes, shifting standards are deployed for female candidates who need to perform better to be assessed equally to male candidates (Van den Brink and Benschop 2012, O’Connor and O’Hagan 2015, Husu and Koskinen 2010, Van den Brink 2016, Foschi 2004, 1996, Biernat 2012, Moss-Racusin et al. 2012, Ridgeway 2011). Although indicators to construct excellence have been discussed as not being gender neutral themselves: Output indicators like the number of publications/citations reproduce gender imbalances: the lower amount of time women are able to spend because of care responsibilities and unpaid work (Xie and Shauman 1998) lowers their productivity (van den Brink and Benschop 2012) as well as the hierarchical position: women are generally lower positioned and publish less (Aksnes et al. 2011).
In our analysis we found further arguments how seemingly neutral indicators might reflect women’s excellence less: Using ‘lone publications’ as indicator for a PI’s independence – one of the ERC elements of excellence – it was argued that having such a publication can be much more difficult for a (typical) female applicant. When supervisors try to hinder their post-docs to publish without them to avoid further competition, women might be (perceived as) less able or willing to fight for it: “Perhaps women think that they don’t want to work against their former boss. And men think: ‘I will show him!’” (Panel member 1, female). When women in fact fight less for having a lone publication, they seem less excellent. The same indicator applied equally to female and male applicants may reproduce inequalities from the science system as it does not reflect the excellence of female and male applicants equally.
But not only are indicators for constructing excellence gendered themselves, they are also perceived as gendered in the evaluation process: We found some empirical evidence on gender bias due to gender stereotypes which determine how indicators are evaluated. Again discussing ‘lone publications’, it was obvious that male applicants are perceived as better fitting to the ideal of an excellent researcher, as male applicants are perceived as more willing to become independent and more motivated to have a publication without the supervisor. In contrast, female applicants are portrayed as not willing to become independent: “In my experience, women are much more satisfied when they can collaborate with someone they know well, that gives safety, while men have the ambition to kick off, to start their own thing as early as possible” (Panel member 31, male). This panel member’s evaluation practices are guided by gender stereotypes which mostly are unconscious, and by this even more guiding.
5 Excellence – gender practices
Some practices in the evaluation process affect female and male applicants differently, what is called gender practices.
One practice we identified is that elements of excellence are dropped for male applicants. We found evidence that indicators like independence and mobility are sometimes deployed for women, but dropped for men. “Having produced at least one important publication without the participation of their PhD supervisor” is formalised in the ERC work programme as an indicator for the applicant’s independence which is one of the ERC elements of excellence. Mobility is not formalised, but emerged as such during the interviews with panel members.
“It is sometimes observed that independence is an issue for women only and Panel members] tend to oversee it for males” (Panel member 12, female). Female applicants who are not able to prove independence are blamed for this lack of independence. But male applicants who might as well not meet this guideline are not checked on this and are therefore advantaged. “And the males come out much better than the females in that aspect” (Panel member 19, female).
The same panel practice is very well illustrated for another aspect of independence, the geographic independence, discussed also as mobility. Female applicants who are not able to prove mobility are penalised while male applicants are not: “[...] for women it is sometimes mentioned that she didn’t go abroad for her PhD or after her PhD.” While at the same time, men’s mobility is not questioned: “[...] men have never moved out of their university (...). And they also become professors at the same university. And everybody finds that they have an excellent CV” (Panel member 32, female). Male applicants who are less independent and/or mobile are not perceived as less excellent whereas female applicants are. This gender practice of questioning women only (deploying double standards) favours men by masking their weaknesses.
In both cases, the indicators men are not questioned on are perceived as typical masculine attributes – independent and mobile, so the individual male applicant benefits from established gender stereotypes. Women as a social category are perceived as less fitting to the ideal of the “independent academic” what clearly has an impact on the assessment of an individual female applicant.
Another gendered practice is related to “ability for ground-breaking research as an element of ERC excellence.” To demonstrate the ground-breaking character of research ‘Selling science’ has been discussed in the interviews with panel members as an important ability for successfully applying for an ERC StG. Also applicants mentioned it as a requirement, referring mainly to the coaching sessions where they have been taught how to sell science. In this way, ‘selling science’ emerged as an informal indicator for assessing excellence. Panel members described this form of self-confident presentation, already in the written proposal, as more common for male applicants; they write their proposal well, stressing very much the strengths and innovativeness and by this practicing a form of impression management. In contrast, proposals of female applicants in some panels were perceived as more modest in step 1, by this risking not being successful in getting to step 2. But in some panels overselling is uncovered and the less exaggerating proposals of female researchers – (“females write more sound and realistic, and down to earth” Panel member 12, female) – bring a benefit in step 2, when men are exposed when overselling their research idea or their personal performance. In contrast, in panels appreciating overselling and taking it as a normative standard, female applicants who are not selling enough are perceived as deficient from the norm set up by most male candidates. Women are seen as “shyer and [...] not convincing people of this being the best idea” (Panel member 28, female). Gender stereotypes become an explaining factor: attributes relevant for selling science like assertiveness – generally labelled as masculine – are taken as the norm and male performance is seen as the standard against which female performance is measured. But compared to men, the range for successful behaviour is narrower for female applicants, because when acting too much along this male norm, they are quickly perceived as too confident or even aggressive, because they violate gender stereotypes to be modest and uncompetitive. Women who do not sell their research in line with gender norms may be blamed for it, but women who are not selling in the traditional male way are disfavoured, too. Selling science in a competitive way is a classic double bind for women. Again, different standards are applied for male and female applicants.
That men and women are treated differently was also supported by a linguistic analysis of the review reports. After controlling for past performance, we found that in the reviews of applications of female researchers, more negation words were used. This suggests an on average more critical attitude towards female applicants than towards male (D6.1).
6 Gender competence and female panel members
It has been demonstrated so far how gender may play a role in the ERC peer review process, on the way how indicators and elements of excellence are gendered and how they are deployed differently to female and male applicants by panel members. This brings in the question of gender awareness in the process.
In interviews some ERC staff members (Scientific Officers) asked for relevant findings from research on gender in science and demonstrated interest in increasing their gender knowledge. Interviewing panel members, it became evident that gender awareness and gender competence exist only marginally. While some panel members argue that gender is no issue in the evaluation of ERC grants and only excellence counts, others take well into account that the ERC pays attention to gender equality, which is reflected in gender action plans, in eligibility criteria and in gender statistics, which are shown in briefing sessions before panel meetings. Some panel members refer to the ERC’s constant aim to try to have equal success rates for female and male applicants, which indicates a certain level of gender awareness, but no clear instructions for implementing them are provided by ERC.
The ERC equality policy goes for a higher number of women in peer review panels, in line with the gender equality targets of the European Research Area (ERA) aiming to increase the share of women in decision making bodies. But a higher share of female panel members so far has not led to a higher share of female grantees, we even found a bivariate negative correlation (D4.1 D6.1): more female panellists go together with lower female success rates (D4.1). Moving to a more complex multivariate analysis, this effect remains, and we find positive effects of the overall success rate, and of the diversity (country wise) in the panel (D6.1).
When female panel members are expected to bring about change, this would imply that the formal framework, assessment criteria for researcher (PI) or research (project) would in practice be deployed differently by female panel members in the evaluation process. One argument in this context is that men and women make sense of whatever they do, based on personal experiences (Ridgeway 2009). So more female panel members bring in more experiences from women, independently if those were perceived as positive or negative. Heilman (2012) argues that women evaluators typically show similar levels of gender bias compared to men, which is presumably a combination of holding similar stereotypes and their own life experiences. When talking about the lower number of female grantees, panel members in fact often referred to gendered experiences in their own career, also to observations they made in their personal research context or just to gender stereotypes as general attributions how men and women are. But it is important to notice that in the ERC assessment process active actions to fight gender bias were reported from female panel members only. Nevertheless, women as evaluators tend to be in line with the standard norms established by the system they represent. Female ERC panel members also want to be perceived and respected as fully committed to the ERC rules and excellence concept, so this would be an argument why there should not be a difference to male panel members and they would not contribute to increase success rates of female applicants.
7 Sub-optimal peer review process
Beside gender-related aspects like described above, also general, not gender-related evaluation practices could be identified that are relevant to explain differences in success rates. They provide insights in the routines and challenges of ERC grant selection and also serve as a basis to understand gendered practices. Performance indicators only explain about half of the variance of scores. So, beside the definition of elements of excellence, the deployment of them in general and from a gender perspective is relevant for explaining lower success rates of female applicants.
• Lack of well-defined indicators and individual operationalisation:
As no appropriate indicators for measuring the different elements of excellence are defined, panel members often select and discuss the operationalisation (= assessing in the remote phase in step 1 and step 2) without linking guidelines to a specific indicator of excellence, e.g. top publications, and use them individually. Analysing the scores demonstrates the panel members’ problems to distinguish between criteria: correlation between the score for the PI and for the project is about 90% (D5.1). This is even more relevant when the topic of the proposal is not in the panel member’s core field of expertise, e.g. that panel members lack expertise how to assess originality. Some panel members then look for external sources of information, which takes time and is not formally correct. Therefore, better-defined indicators would support the evaluation process.
• Non-binding guidelines
When several ERC documents give guidelines to operationalise ERC elements of evaluation, not all panel members deploy them or deploy them in the same way. This is exemplified by the criterion of “independence”: The applicant’s independence is demonstrated by having “at least one important publication without the PhD supervisor”. But this guideline is not fully taken into account, some panel members don’t consider it at all, but mentioned other indicators which are not formalised in ERC documents like gaining funding budget, the composition of the team lead by the applicant, topical distance to the supervisor (D5.1) working in different home institutions or establishing new research collaborations. In this way, very different achievements are assessed when operationalising a specific criterion.
• Deploying guidelines selectively
Formal elements of excellence are not deployed systematically; different panel members focus on different elements of excellence: for some panel members, the research idea is most important, for others it is the track record. Some construct excellence based on a ‘Nature’ publication only while others refer strongly to a PI’s continuous research line including publications in field specific journals. By many panel members, publications are the core indicator for describing excellence in ERC StG selection. But sometimes the quality of the journal, sometimes the impact scores, the number of publications or whether the applicant is first, last or single authors is relevant. Quite often publications in high ranked journals (short: top publications) like ‘Nature’ or ‘Science’ were pointed out as a ‘kind of core excellence’ in LS panels. The regression analysis in D6.1 clearly shows this. Research has shown that different standards about the relevance of particular criteria in the evaluation process may be gendered, as evaluators tend to overstate/understate the relevance of a criterion depending on whether it is attributed to a male or a female applicant (Uhlmann and Cohen 2005, Baltes and Parker 2000).
• Weighing PI versus research project
In the interviews it was widely discussed how much weight should be given to which element of excellence. There seems to be a kind of uncertainty among the panel members. The question of weighing assessments of PI versus research project was also raised. Some panel members find a “Nature” publication a clear and sufficient indicator for excellence and don’t consider the quality of the proposal, which might not be excellent at all. In other cases, the PI’s profile is more or less neglected. This is also illustrated by the fact that the correlation between the score for the PI and the score for the project is high: in step 1 almost 0.90 in step 2 about 0.60 (D5.1). This suggests that the reviewers make no difference between PI and project, although these are rather different dimensions of excellence. It implies that in step 1 the PI score is dominant, and that the project score simply follows. From a gender perspective, the pattern is not the same for men and women, but the patterns differ between the domains. In LS, where we found most gender bias, women are more assessed in terms of the project score, and men more in terms of the PI score. In the PE domain, it is the other way around (D5.1).
In step 2, the correlation between the two scores is still high but not as high as in step 1. When predicting the final score, the PI score has no effect at all anymore. These findings support the different views expressed in the interviews. Both is correct (emphasis on the PI’s CV and emphasis on the project), but in different phases of the evaluation (D5.1).
• Deploying informal elements of excellence
A further general practice of constructing excellence is to deploy informal elements of excellence, which may emerge during the panel discussions to highlight specific strengths or weaknesses of one special application. As these strengths or weaknesses are not reviewed for other applications, this might imply an evaluation bias. The most mentioned informal elements of evaluation are mobility, the (prestige of the) host institution, collaboration networks, the aim to bring researchers back-to-Europe (from US)and personal characteristics like assertiveness, the well-written nature of the proposal and the ability to sell it.
• Rejecting formal elements of the evaluation
For some panel members the formal elements of excellence set up by the ERC are only of (very) limited relevance. They have in mind a clear concept of excellence (irrespective of the specific ERC definition) and they explain their ability to recognise excellence by their experience and describe it as “a kind of a gut feeling” (Panel member 18, male). Those panel members’ intuition may in some cases be valid predictors of applicants’ success. However, overconfidence in one’s ability to recognise excellence – “as experienced evaluators we know what to do” (Panel member 29, male) – reduces validity and increases the risk of suboptimal decisions. These panel members demonstrate that the construction of excellence is hardly related to the formalised ERC criteria.
A few panel members fully reject the formalisation of excellence at all and see it as a way “to kill scientific excellence” (Panel member 18, male).
Other panel members reject some parts of the excellence concept currently in practice: Independence as an element of excellence in ERC grant selection encourages to foster an attitude that focuses on the individual achievements and merits and at the same time ignores other colleagues’ contributions. Therefore, the researcher’s ability to collaborate and encourage or guide other researchers may be neglected. This can be seen as a kind of bias as research conducted by one researcher only can hardly be found in the LS field. Some panel members argued that in average, this is more difficult for women who tend to rely more on team contributions and efforts. Women also might feel more resistance to put only themselves as authors and not mention colleagues who also have contributed to the research and/or the publication: “Having to sell this entire team project as if it was my own, as if I was a ‘leader’ when in fact I’m always collaborating and learning from everyone” (Applicant, female).
Based on a lack of gender awareness and competence (see 6) these suboptimal practices in the peer review process also lead to gender practices (see 5).
Potential Impact:
Potential Impact is on the improvement of Project selection procedures and mechanisms´- also beyond the ERC, as the ERC is generally seen as 'role model'. More specifically our study resulted in the following recommendations.
When analysing potential sources for gender bias in the ERC peer review process, we found that gender bias exists (after controlling for the PI’s past performance) and that it can be located on two different levels. Evidence-based recommendations for improvements have been developed on both levels.
As we identified some general suboptimal evaluation practices – due to a vague definition of excellence and a quite open process how to apply different elements of excellence in practice – it is recommended to (1) more specifically define what excellence means in the context of ERC as a prestigious funder for blue sky / frontier research (e.g. at the panel level). To (2) better standardise the process will reduce space for individual interpretations and improve the selection process.
Such interventions would also help to reduce bias on a second level: gendered practices – which affect female and male applicants differently – have been identified. Treating men and women differently is mostly done unconsciously and related to personal assumptions and attributions on how men and women are or should be (=gender stereotypes). When in some panels independence or mobility are checked for women only while they are not questioned for males – for the latter, they seem naturally given – male applicants are favoured, different standards are applied. To change this pattern, room for reflections is needed to (3) raise awareness for gendered practices. Therefore, unconscious gender bias checks or trainings, films and more can be provided and offered to panel members, ERCEA staff and also the ERC as a funding institution in an efficient format. By (4) recognising that gender plays a role in the peer review process and when constructing excellence, ERC could take over a pioneer role for more gender equality in science and for better selecting the best (female and male) applicants.
We hope that our evidence-based arguments contribute to better understand what is going on in the ERC peer-review process and in panel meetings and to illustrate where and how the quality of the peer-review process in general and gender fairness in the evaluation process can be optimised.
List of Websites:
No website
To explain lower success rates of female applicants in ERC grants, we collected data about past performance of the applicants and interviewed panel members about how selection criteria are practiced in general and specifically for female vs. male applicants. Controlling for past performance, we found gender bias. The analysis of the interviews provides empirical evidence that current evaluation practices are suboptimal, leading to some gender-biased (gendered) practices. Providing empirical evidence of gendered practices raises gender awareness and gives a chance to modify and improve them.
When analysing potential sources for gender bias in the ERC peer review process, we found that gender bias exists (after controlling for the PI’s past performance) and that it can be located on two different levels. Evidence-based recommendations for improvements have been developed on both levels.
As we identified some general suboptimal evaluation practices – due to a vague definition of excellence and a quite open process how to apply different elements of excellence in practice – it is recommended to (1) more specifically define what excellence means in the context of ERC as a prestigious funder for blue sky / frontier research (e.g. at the panel level). To (2) better standardise the process will reduce space for individual interpretations and improve the selection process.
Such interventions would also help to reduce bias on a second level: gendered practices – which affect female and male applicants differently – have been identified. Treating men and women differently is mostly done unconsciously and related to personal assumptions and attributions on how men and women are or should be (=gender stereotypes). When in some panels independence or mobility are checked for women only while they are not questioned for males – for the latter, they seem naturally given – male applicants are favoured, different standards are applied. To change this pattern, room for reflections is needed to (3) raise awareness for gendered practices. Therefore, unconscious gender bias checks or trainings, films and more can be provided and offered to panel members, ERCEA staff and also the ERC as a funding institution in an efficient format. By (4) recognising that gender plays a role in the peer review process and when constructing excellence, ERC could take over a pioneer role for more gender equality in science and for better selecting the best (female and male) applicants.
We hope that our evidence-based arguments contribute to better understand what is going on in the ERC peer-review process and in panel meetings and to illustrate where and how the quality of the peer-review process in general and gender fairness in the evaluation process can be optimised.
Project Context and Objectives:
In this project, we investigate the extent and sources of gender bias in grant reviewing schemes, using ERC Starting Grants and Advanced Grants as case studies. The project focuses on the three main aspects of the schemes: the formal requirements and procedures, the composition of the reviewers and panels, and the process of application and evaluation. More specifically the following questions are answered: (i) Does the application process lead to gendered self-selection? (ii) Where in the selection process are (potentially) gender-biased decisions made? (iii) Are the decision criteria gender-biased, and/or are they deployed in a gender-biased manner? (iv) To what extent is this due to the (gender or topical biased) composition of the panels and/or reviewers?
This project has two general objectives:
(1) To identify potential (in-)direct discrimination in institutional processes and practices of ERC funding activities due to the gender dimension
(2) To analyse the causes of gender bias in grant allocation, in order to support gender-neutral selection processes at the ERC level, and as an example for other funding agencies
More specific objectives are targeted in the different work packages:
(a) To identify potential biases in applying for ERC Starting Grants and Advanced Grants
(b) To clarify the dimensions of the concept of excellence as it is de facto used in peer review and panel review (WP2, WP3)
(c) To identify the gendered elements in the conception of excellence, that is those elements that work negatively for women (WP2, WP3, WP5)
(d) To identify gendered practices of deploying the excellence dimensions (WP5)
(e) To determine the size of gender bias in ERC grant decisions, and contributing to a reliable and valid method for measuring gender bias. (WP6, WP7)
(f) To contribute to a gender-neutral excellence concept and to procedures of a gender neutral way of deploying the excellence criteria in practice (WP5, WP7)
(g) To identify the phases in the review process where gender bias increases and/or decreases (WP6)
(h) To clarify the process of peer reviewer and panellist selection, and to identify the gendered and topical bias in the group of reviewers and the group of panellists (WP4)
(i) To understand the causes of gender bias, and especially the role of
• gendered criteria (WP3)
• gendered application of neutral criteria (WP5, WP4)
• gender-biased panels (WP4, WP5, WP6)
• topical-biased panels (WP6)
(j) Proposing a peer and panel review procedure that is not gender-biased (WP7)
Project Results:
1 Context: Success rates in ERC panels
Starting point for our study was the fact that in all ERC grants female applicants have lower success rates (relation applicants/grantees) than men, especially in the LS domain. We have looked at all nine LS panels of the StG 2014 and analysed the success rates in step 1 and step 2 of the evaluation process – since differences can be located more accurately and potential improvements can be installed more effectively. In this case, women have a 6% lower success rate in step 1 and 2% in step 2, which makes an overall negative difference of 3.
2 Methods and data
Apart from controlling past performance we used a mix of quantitative and qualitative data to provide insight in the nature and prevalence of the bias. Our findings are based mainly on qualitative interviews conducted with 32 members of ERC peer review panels. Additionally, we analysed ERC policy documents and sent an online survey to all applicants who had given informed consent to be contacted. More than 1.200 answered the questionnaire. Further sources of data were evaluation reports from panel members and external evaluators, CVs of applicants, and Web-of-Science data for about 40% of the applicants and for panel members of four panels. We used the data to test our model, in which success is explained by performance (quality) of the applicant and gender.
In the summary below, we refer to the deliverable when mentioning our quantitative analyses. When citing interview data or other literature, this becomes clear through quotes and references.
3 What is gender bias?
Gender differences are not the same as gender bias. In order to preclude the possibility that different success rates are due to differences in past performance, we collected the performance data for all applicants of 4 panels, and about 30% of the applicants of the other 5 panels. Of the four panels, three had similar patterns and the fourth was rather different. When controlling for past performance, gender remained having a significant effect on success, in favour of men. In the fourth panel, it was exactly the other way around. At the LS-domain level, the gender bias (in favour of men) is still visible, but non-significant. For PE and SH panels we only have the results at domain level, SH is similar to LS. The following conclusions can be drawn:
- Gender bias exists.
- It exists in two directions, but more often in favour of men.
- At field/domain level, the opposite effects partly even out; the remainder effect still is in favour of men (LS and SH), but less strong.
- In PE, gender effect at the domain level is about absent.
Now, how can this gender bias in the evaluation procedure be explained? For that we did some quantitative analyses and heavily drew on the analysis of the interviews with panel members. Finally, we again emphasise that observations are essential.
4 Excellence – gendered indicators
As ‘Excellence’ is claimed to be the core criterion in the evaluation process of ERC frontier research grants (see Work programme 2014: p. 20), the understanding and operationalisation of excellence were a major analytical step in our study.
So far, mostly quantitative metrics function as criteria for excellence to demonstrate an applicant’s scientific performance and success. Publications and impact points have become of increasing relevance and reflect the meritocratic understanding that quality of science can be measured by transparent and fair indicators. Yet, it has been widely demonstrated in recent research that the concept of excellence and the criteria for excellence are socially constructed and not gender neutral. Male-dominated networks are relevant in recruitment processes, shifting standards are deployed for female candidates who need to perform better to be assessed equally to male candidates (Van den Brink and Benschop 2012, O’Connor and O’Hagan 2015, Husu and Koskinen 2010, Van den Brink 2016, Foschi 2004, 1996, Biernat 2012, Moss-Racusin et al. 2012, Ridgeway 2011). Although indicators to construct excellence have been discussed as not being gender neutral themselves: Output indicators like the number of publications/citations reproduce gender imbalances: the lower amount of time women are able to spend because of care responsibilities and unpaid work (Xie and Shauman 1998) lowers their productivity (van den Brink and Benschop 2012) as well as the hierarchical position: women are generally lower positioned and publish less (Aksnes et al. 2011).
In our analysis we found further arguments how seemingly neutral indicators might reflect women’s excellence less: Using ‘lone publications’ as indicator for a PI’s independence – one of the ERC elements of excellence – it was argued that having such a publication can be much more difficult for a (typical) female applicant. When supervisors try to hinder their post-docs to publish without them to avoid further competition, women might be (perceived as) less able or willing to fight for it: “Perhaps women think that they don’t want to work against their former boss. And men think: ‘I will show him!’” (Panel member 1, female). When women in fact fight less for having a lone publication, they seem less excellent. The same indicator applied equally to female and male applicants may reproduce inequalities from the science system as it does not reflect the excellence of female and male applicants equally.
But not only are indicators for constructing excellence gendered themselves, they are also perceived as gendered in the evaluation process: We found some empirical evidence on gender bias due to gender stereotypes which determine how indicators are evaluated. Again discussing ‘lone publications’, it was obvious that male applicants are perceived as better fitting to the ideal of an excellent researcher, as male applicants are perceived as more willing to become independent and more motivated to have a publication without the supervisor. In contrast, female applicants are portrayed as not willing to become independent: “In my experience, women are much more satisfied when they can collaborate with someone they know well, that gives safety, while men have the ambition to kick off, to start their own thing as early as possible” (Panel member 31, male). This panel member’s evaluation practices are guided by gender stereotypes which mostly are unconscious, and by this even more guiding.
5 Excellence – gender practices
Some practices in the evaluation process affect female and male applicants differently, what is called gender practices.
One practice we identified is that elements of excellence are dropped for male applicants. We found evidence that indicators like independence and mobility are sometimes deployed for women, but dropped for men. “Having produced at least one important publication without the participation of their PhD supervisor” is formalised in the ERC work programme as an indicator for the applicant’s independence which is one of the ERC elements of excellence. Mobility is not formalised, but emerged as such during the interviews with panel members.
“It is sometimes observed that independence is an issue for women only and Panel members] tend to oversee it for males” (Panel member 12, female). Female applicants who are not able to prove independence are blamed for this lack of independence. But male applicants who might as well not meet this guideline are not checked on this and are therefore advantaged. “And the males come out much better than the females in that aspect” (Panel member 19, female).
The same panel practice is very well illustrated for another aspect of independence, the geographic independence, discussed also as mobility. Female applicants who are not able to prove mobility are penalised while male applicants are not: “[...] for women it is sometimes mentioned that she didn’t go abroad for her PhD or after her PhD.” While at the same time, men’s mobility is not questioned: “[...] men have never moved out of their university (...). And they also become professors at the same university. And everybody finds that they have an excellent CV” (Panel member 32, female). Male applicants who are less independent and/or mobile are not perceived as less excellent whereas female applicants are. This gender practice of questioning women only (deploying double standards) favours men by masking their weaknesses.
In both cases, the indicators men are not questioned on are perceived as typical masculine attributes – independent and mobile, so the individual male applicant benefits from established gender stereotypes. Women as a social category are perceived as less fitting to the ideal of the “independent academic” what clearly has an impact on the assessment of an individual female applicant.
Another gendered practice is related to “ability for ground-breaking research as an element of ERC excellence.” To demonstrate the ground-breaking character of research ‘Selling science’ has been discussed in the interviews with panel members as an important ability for successfully applying for an ERC StG. Also applicants mentioned it as a requirement, referring mainly to the coaching sessions where they have been taught how to sell science. In this way, ‘selling science’ emerged as an informal indicator for assessing excellence. Panel members described this form of self-confident presentation, already in the written proposal, as more common for male applicants; they write their proposal well, stressing very much the strengths and innovativeness and by this practicing a form of impression management. In contrast, proposals of female applicants in some panels were perceived as more modest in step 1, by this risking not being successful in getting to step 2. But in some panels overselling is uncovered and the less exaggerating proposals of female researchers – (“females write more sound and realistic, and down to earth” Panel member 12, female) – bring a benefit in step 2, when men are exposed when overselling their research idea or their personal performance. In contrast, in panels appreciating overselling and taking it as a normative standard, female applicants who are not selling enough are perceived as deficient from the norm set up by most male candidates. Women are seen as “shyer and [...] not convincing people of this being the best idea” (Panel member 28, female). Gender stereotypes become an explaining factor: attributes relevant for selling science like assertiveness – generally labelled as masculine – are taken as the norm and male performance is seen as the standard against which female performance is measured. But compared to men, the range for successful behaviour is narrower for female applicants, because when acting too much along this male norm, they are quickly perceived as too confident or even aggressive, because they violate gender stereotypes to be modest and uncompetitive. Women who do not sell their research in line with gender norms may be blamed for it, but women who are not selling in the traditional male way are disfavoured, too. Selling science in a competitive way is a classic double bind for women. Again, different standards are applied for male and female applicants.
That men and women are treated differently was also supported by a linguistic analysis of the review reports. After controlling for past performance, we found that in the reviews of applications of female researchers, more negation words were used. This suggests an on average more critical attitude towards female applicants than towards male (D6.1).
6 Gender competence and female panel members
It has been demonstrated so far how gender may play a role in the ERC peer review process, on the way how indicators and elements of excellence are gendered and how they are deployed differently to female and male applicants by panel members. This brings in the question of gender awareness in the process.
In interviews some ERC staff members (Scientific Officers) asked for relevant findings from research on gender in science and demonstrated interest in increasing their gender knowledge. Interviewing panel members, it became evident that gender awareness and gender competence exist only marginally. While some panel members argue that gender is no issue in the evaluation of ERC grants and only excellence counts, others take well into account that the ERC pays attention to gender equality, which is reflected in gender action plans, in eligibility criteria and in gender statistics, which are shown in briefing sessions before panel meetings. Some panel members refer to the ERC’s constant aim to try to have equal success rates for female and male applicants, which indicates a certain level of gender awareness, but no clear instructions for implementing them are provided by ERC.
The ERC equality policy goes for a higher number of women in peer review panels, in line with the gender equality targets of the European Research Area (ERA) aiming to increase the share of women in decision making bodies. But a higher share of female panel members so far has not led to a higher share of female grantees, we even found a bivariate negative correlation (D4.1 D6.1): more female panellists go together with lower female success rates (D4.1). Moving to a more complex multivariate analysis, this effect remains, and we find positive effects of the overall success rate, and of the diversity (country wise) in the panel (D6.1).
When female panel members are expected to bring about change, this would imply that the formal framework, assessment criteria for researcher (PI) or research (project) would in practice be deployed differently by female panel members in the evaluation process. One argument in this context is that men and women make sense of whatever they do, based on personal experiences (Ridgeway 2009). So more female panel members bring in more experiences from women, independently if those were perceived as positive or negative. Heilman (2012) argues that women evaluators typically show similar levels of gender bias compared to men, which is presumably a combination of holding similar stereotypes and their own life experiences. When talking about the lower number of female grantees, panel members in fact often referred to gendered experiences in their own career, also to observations they made in their personal research context or just to gender stereotypes as general attributions how men and women are. But it is important to notice that in the ERC assessment process active actions to fight gender bias were reported from female panel members only. Nevertheless, women as evaluators tend to be in line with the standard norms established by the system they represent. Female ERC panel members also want to be perceived and respected as fully committed to the ERC rules and excellence concept, so this would be an argument why there should not be a difference to male panel members and they would not contribute to increase success rates of female applicants.
7 Sub-optimal peer review process
Beside gender-related aspects like described above, also general, not gender-related evaluation practices could be identified that are relevant to explain differences in success rates. They provide insights in the routines and challenges of ERC grant selection and also serve as a basis to understand gendered practices. Performance indicators only explain about half of the variance of scores. So, beside the definition of elements of excellence, the deployment of them in general and from a gender perspective is relevant for explaining lower success rates of female applicants.
• Lack of well-defined indicators and individual operationalisation:
As no appropriate indicators for measuring the different elements of excellence are defined, panel members often select and discuss the operationalisation (= assessing in the remote phase in step 1 and step 2) without linking guidelines to a specific indicator of excellence, e.g. top publications, and use them individually. Analysing the scores demonstrates the panel members’ problems to distinguish between criteria: correlation between the score for the PI and for the project is about 90% (D5.1). This is even more relevant when the topic of the proposal is not in the panel member’s core field of expertise, e.g. that panel members lack expertise how to assess originality. Some panel members then look for external sources of information, which takes time and is not formally correct. Therefore, better-defined indicators would support the evaluation process.
• Non-binding guidelines
When several ERC documents give guidelines to operationalise ERC elements of evaluation, not all panel members deploy them or deploy them in the same way. This is exemplified by the criterion of “independence”: The applicant’s independence is demonstrated by having “at least one important publication without the PhD supervisor”. But this guideline is not fully taken into account, some panel members don’t consider it at all, but mentioned other indicators which are not formalised in ERC documents like gaining funding budget, the composition of the team lead by the applicant, topical distance to the supervisor (D5.1) working in different home institutions or establishing new research collaborations. In this way, very different achievements are assessed when operationalising a specific criterion.
• Deploying guidelines selectively
Formal elements of excellence are not deployed systematically; different panel members focus on different elements of excellence: for some panel members, the research idea is most important, for others it is the track record. Some construct excellence based on a ‘Nature’ publication only while others refer strongly to a PI’s continuous research line including publications in field specific journals. By many panel members, publications are the core indicator for describing excellence in ERC StG selection. But sometimes the quality of the journal, sometimes the impact scores, the number of publications or whether the applicant is first, last or single authors is relevant. Quite often publications in high ranked journals (short: top publications) like ‘Nature’ or ‘Science’ were pointed out as a ‘kind of core excellence’ in LS panels. The regression analysis in D6.1 clearly shows this. Research has shown that different standards about the relevance of particular criteria in the evaluation process may be gendered, as evaluators tend to overstate/understate the relevance of a criterion depending on whether it is attributed to a male or a female applicant (Uhlmann and Cohen 2005, Baltes and Parker 2000).
• Weighing PI versus research project
In the interviews it was widely discussed how much weight should be given to which element of excellence. There seems to be a kind of uncertainty among the panel members. The question of weighing assessments of PI versus research project was also raised. Some panel members find a “Nature” publication a clear and sufficient indicator for excellence and don’t consider the quality of the proposal, which might not be excellent at all. In other cases, the PI’s profile is more or less neglected. This is also illustrated by the fact that the correlation between the score for the PI and the score for the project is high: in step 1 almost 0.90 in step 2 about 0.60 (D5.1). This suggests that the reviewers make no difference between PI and project, although these are rather different dimensions of excellence. It implies that in step 1 the PI score is dominant, and that the project score simply follows. From a gender perspective, the pattern is not the same for men and women, but the patterns differ between the domains. In LS, where we found most gender bias, women are more assessed in terms of the project score, and men more in terms of the PI score. In the PE domain, it is the other way around (D5.1).
In step 2, the correlation between the two scores is still high but not as high as in step 1. When predicting the final score, the PI score has no effect at all anymore. These findings support the different views expressed in the interviews. Both is correct (emphasis on the PI’s CV and emphasis on the project), but in different phases of the evaluation (D5.1).
• Deploying informal elements of excellence
A further general practice of constructing excellence is to deploy informal elements of excellence, which may emerge during the panel discussions to highlight specific strengths or weaknesses of one special application. As these strengths or weaknesses are not reviewed for other applications, this might imply an evaluation bias. The most mentioned informal elements of evaluation are mobility, the (prestige of the) host institution, collaboration networks, the aim to bring researchers back-to-Europe (from US)and personal characteristics like assertiveness, the well-written nature of the proposal and the ability to sell it.
• Rejecting formal elements of the evaluation
For some panel members the formal elements of excellence set up by the ERC are only of (very) limited relevance. They have in mind a clear concept of excellence (irrespective of the specific ERC definition) and they explain their ability to recognise excellence by their experience and describe it as “a kind of a gut feeling” (Panel member 18, male). Those panel members’ intuition may in some cases be valid predictors of applicants’ success. However, overconfidence in one’s ability to recognise excellence – “as experienced evaluators we know what to do” (Panel member 29, male) – reduces validity and increases the risk of suboptimal decisions. These panel members demonstrate that the construction of excellence is hardly related to the formalised ERC criteria.
A few panel members fully reject the formalisation of excellence at all and see it as a way “to kill scientific excellence” (Panel member 18, male).
Other panel members reject some parts of the excellence concept currently in practice: Independence as an element of excellence in ERC grant selection encourages to foster an attitude that focuses on the individual achievements and merits and at the same time ignores other colleagues’ contributions. Therefore, the researcher’s ability to collaborate and encourage or guide other researchers may be neglected. This can be seen as a kind of bias as research conducted by one researcher only can hardly be found in the LS field. Some panel members argued that in average, this is more difficult for women who tend to rely more on team contributions and efforts. Women also might feel more resistance to put only themselves as authors and not mention colleagues who also have contributed to the research and/or the publication: “Having to sell this entire team project as if it was my own, as if I was a ‘leader’ when in fact I’m always collaborating and learning from everyone” (Applicant, female).
Based on a lack of gender awareness and competence (see 6) these suboptimal practices in the peer review process also lead to gender practices (see 5).
Potential Impact:
Potential Impact is on the improvement of Project selection procedures and mechanisms´- also beyond the ERC, as the ERC is generally seen as 'role model'. More specifically our study resulted in the following recommendations.
When analysing potential sources for gender bias in the ERC peer review process, we found that gender bias exists (after controlling for the PI’s past performance) and that it can be located on two different levels. Evidence-based recommendations for improvements have been developed on both levels.
As we identified some general suboptimal evaluation practices – due to a vague definition of excellence and a quite open process how to apply different elements of excellence in practice – it is recommended to (1) more specifically define what excellence means in the context of ERC as a prestigious funder for blue sky / frontier research (e.g. at the panel level). To (2) better standardise the process will reduce space for individual interpretations and improve the selection process.
Such interventions would also help to reduce bias on a second level: gendered practices – which affect female and male applicants differently – have been identified. Treating men and women differently is mostly done unconsciously and related to personal assumptions and attributions on how men and women are or should be (=gender stereotypes). When in some panels independence or mobility are checked for women only while they are not questioned for males – for the latter, they seem naturally given – male applicants are favoured, different standards are applied. To change this pattern, room for reflections is needed to (3) raise awareness for gendered practices. Therefore, unconscious gender bias checks or trainings, films and more can be provided and offered to panel members, ERCEA staff and also the ERC as a funding institution in an efficient format. By (4) recognising that gender plays a role in the peer review process and when constructing excellence, ERC could take over a pioneer role for more gender equality in science and for better selecting the best (female and male) applicants.
We hope that our evidence-based arguments contribute to better understand what is going on in the ERC peer-review process and in panel meetings and to illustrate where and how the quality of the peer-review process in general and gender fairness in the evaluation process can be optimised.
List of Websites:
No website