Periodic Reporting for period 1 - AIRE (AIRE_ Artificial Intelligence for Recruiting Europe +)
Reporting period: 2019-10-01 to 2021-02-28
Explanation of the work carried out by the beneficiaries and overview of progress
In a globally connected world of Big Data, the business landscape is evolving rapidly. The HR function is shifting from being a purely administrative function that gathers and reports on data, to making sense of that data, analysing it and using it to inform business strategy. In the case of large/global organisations, the challenges faced i.e. dispersed workforce, non-aligned business processes, re-structuring or outsourcing, there is an evident need to adopt business-wide data-analysis, alignment and integration. As reported by early adopters of AI in Human Resource Management (HRM), traditional CVs have long outstayed their use for employers and overburden HR and email systems. Artificial Intelligence (AI) and Big Data, combined with strategic insight create new business opportunities and transform the way the Human Resource function contributes to companies´ performance and competitive advantage.
The SME has planned and experimented with harnessing the power of natural language processing to deliver real change in the HR domain and has demonstrated real advances in the State-of-Art with the work done during this project. This was the perfect alignment of commercial, technical and experimental competencies with the SME, Project Sponsor and the Innovation Associate to deliver on the objectives which are building blocks to provide users with digitized disruption in a traditionally slow adopter which is the HR domain.
In a globally connected world of Big Data, the business landscape is evolving rapidly. The HR function is shifting from being a purely administrative function that gathers and reports on data, to making sense of that data, analysing it and using it to inform business strategy. In the case of large/global organisations, the challenges faced i.e. dispersed workforce, non-aligned business processes, re-structuring or outsourcing, there is an evident need to adopt business-wide data-analysis, alignment and integration. As reported by early adopters of AI in Human Resource Management (HRM), traditional CVs have long outstayed their use for employers and overburden HR and email systems. Artificial Intelligence (AI) and Big Data, combined with strategic insight create new business opportunities and transform the way the Human Resource function contributes to companies´ performance and competitive advantage.
The SME has planned and experimented with harnessing the power of natural language processing to deliver real change in the HR domain and has demonstrated real advances in the State-of-Art with the work done during this project. This was the perfect alignment of commercial, technical and experimental competencies with the SME, Project Sponsor and the Innovation Associate to deliver on the objectives which are building blocks to provide users with digitized disruption in a traditionally slow adopter which is the HR domain.
Allsorter CV Optimization process has benchmarked improvements for reducing turnaround time for recruiters. This is achieved by parsing and reformatting resumes on runtime thus eliminating the need for using manual transformation. This objective pertains to highlight any differences between original CV and reformatted CV using an ensemble of advanced text similarity methods.
Text Extraction - Gaps Assessment and Detection
A minimum threshold was set for similarity score. Any text in the source not matching any non-aligned text in the target with a score beyond the threshold is marked as a gap. In our experiments, we found that the optimal similarity threshold wavers between 72% and 80%.
Evaluation: In our test-case scenario, we find that the gap analysis framework achieves an annotation accuracy in excess of 96%. Evaluation was manually carried out on a random sample of 10 CVs. Two employees from BPIS’s QA testing and Data Analysis team were assigned to manually inspect the annotation (gap highlights) created by the gap analysis framework for those 10 CVs. This form of assessment is acceptable since both analysts have HR domain expertise and are very experienced with tasks of this nature. Each analyst is expected to record the errors (wrongly highlighted text) and omissions (texts which should be highlighted but not) and for the CV, and to come up with a score of the accuracy. The system achieves an F1 score of 94 % and an accuracy of 96%. This finding corroborates the production use-case where feedback from commercial users of Allsorter has been overwhelmingly positive.
Potential Bias Information Highlight – AIRE has determined potential bias elements in a CV after interviewing recruiters from multiple organizations. In addition to a person’s ethnic origin, age, and gender, their work experiences in certain organizations or education from certain institutions may also impact on their chances of hiring. This objective will allow identifying and highlighting all such items on a CV that a recruiter may view on the original CV and then have the option of removing these items. To achieve this objective, several subtasks were defined as below.
In order for AIRE to achieve this objective, the developed system must show a capability for language understanding in order to approximate the meaning of skills and job description. For instance, a good system must understand synonymous terms as well as related terms. A very simple example is the skill ‘Java’ which though not similar to the terms ‘Developer’ and ‘Programmer’, are clearly related. Existing systems would fail to match CVs where an exact keyword being searched is missing. This has the potential of missing out on tons of candidates whose skills and experience may be written on their CV differently from the query terms being used by the recruiter.
AIRE has determined that in order to achieve a skills-based CV ranking, it is important to improve the skills extraction pipeline of Allsorter. Allsorter currently extracts some hard and soft skills, classified into 4 classes, namely – Professional, Computer, Language, and Soft skills, however, the current skills extraction system is not optimal. The IA has worked with the company’s domain expert to annotate some CVs and job descriptions. The annotation includes skills as well as their left and right surrounding context. The IA has performed a scaled evaluation of the skill extractor model and has observed an F1 score of 92% with a rough increase of 4% when hard skills alone are considered.
Performance Testing:
The semantic annotation pipeline described earlier was testing using feature ablation and cross fold validation (K=10) resulting in an F-measure of 0.93 (93% precision and recall on test corpus)
Text Extraction - Gaps Assessment and Detection
A minimum threshold was set for similarity score. Any text in the source not matching any non-aligned text in the target with a score beyond the threshold is marked as a gap. In our experiments, we found that the optimal similarity threshold wavers between 72% and 80%.
Evaluation: In our test-case scenario, we find that the gap analysis framework achieves an annotation accuracy in excess of 96%. Evaluation was manually carried out on a random sample of 10 CVs. Two employees from BPIS’s QA testing and Data Analysis team were assigned to manually inspect the annotation (gap highlights) created by the gap analysis framework for those 10 CVs. This form of assessment is acceptable since both analysts have HR domain expertise and are very experienced with tasks of this nature. Each analyst is expected to record the errors (wrongly highlighted text) and omissions (texts which should be highlighted but not) and for the CV, and to come up with a score of the accuracy. The system achieves an F1 score of 94 % and an accuracy of 96%. This finding corroborates the production use-case where feedback from commercial users of Allsorter has been overwhelmingly positive.
Potential Bias Information Highlight – AIRE has determined potential bias elements in a CV after interviewing recruiters from multiple organizations. In addition to a person’s ethnic origin, age, and gender, their work experiences in certain organizations or education from certain institutions may also impact on their chances of hiring. This objective will allow identifying and highlighting all such items on a CV that a recruiter may view on the original CV and then have the option of removing these items. To achieve this objective, several subtasks were defined as below.
In order for AIRE to achieve this objective, the developed system must show a capability for language understanding in order to approximate the meaning of skills and job description. For instance, a good system must understand synonymous terms as well as related terms. A very simple example is the skill ‘Java’ which though not similar to the terms ‘Developer’ and ‘Programmer’, are clearly related. Existing systems would fail to match CVs where an exact keyword being searched is missing. This has the potential of missing out on tons of candidates whose skills and experience may be written on their CV differently from the query terms being used by the recruiter.
AIRE has determined that in order to achieve a skills-based CV ranking, it is important to improve the skills extraction pipeline of Allsorter. Allsorter currently extracts some hard and soft skills, classified into 4 classes, namely – Professional, Computer, Language, and Soft skills, however, the current skills extraction system is not optimal. The IA has worked with the company’s domain expert to annotate some CVs and job descriptions. The annotation includes skills as well as their left and right surrounding context. The IA has performed a scaled evaluation of the skill extractor model and has observed an F1 score of 92% with a rough increase of 4% when hard skills alone are considered.
Performance Testing:
The semantic annotation pipeline described earlier was testing using feature ablation and cross fold validation (K=10) resulting in an F-measure of 0.93 (93% precision and recall on test corpus)
THE CONCLUSION OF THE ACTION
We have recently added value to recruiters and jobseekers with our CV Optimisation system as a direct result of our R & D into the AIRE platform design. We have found that users (B2B) are Recruitment Companies and Large Corporates which reformat and tailor CV’s into a specific template. This is a process being completed across the global marketplace however up to now it is being executed manually. With GDPR and speed delays, the practices of crowdsourcing this is not viable and we have developed and deployed the 1st to market CV Optimiser for this purpose.
Our system uploads a CV in any format, then the user has the ability to edit in an editor screen and reprint the new CV document in less than 10 seconds.
For the consumer(B2C) customer market we have identified another practical deployment of AIRE will be:
1. The ability to download the job application being sought by the jobseeker
2. We then upload their CV and identify where there are gaps
3. Recommend skills they have from the ESCO understanding of their experience to date
4. Create the best representation of the jobseeker’s skills to date
5. Constantly help iterate for the user according to different jobs being applied for
Our conclusion is the CV Optimiser is the quickest route to market within the development lifecycle of the AIRE project and is now earning revenue.
We see a fantastic opportunity to develop the AIRE project in conjunction with this 1st phase product as the ability of our software is enhanced by extracting all the data received from 1,000’s of CV applications ensured our product is further embedded in the market with our machine learning getting real time data for dynamic capability.
We have recently added value to recruiters and jobseekers with our CV Optimisation system as a direct result of our R & D into the AIRE platform design. We have found that users (B2B) are Recruitment Companies and Large Corporates which reformat and tailor CV’s into a specific template. This is a process being completed across the global marketplace however up to now it is being executed manually. With GDPR and speed delays, the practices of crowdsourcing this is not viable and we have developed and deployed the 1st to market CV Optimiser for this purpose.
Our system uploads a CV in any format, then the user has the ability to edit in an editor screen and reprint the new CV document in less than 10 seconds.
For the consumer(B2C) customer market we have identified another practical deployment of AIRE will be:
1. The ability to download the job application being sought by the jobseeker
2. We then upload their CV and identify where there are gaps
3. Recommend skills they have from the ESCO understanding of their experience to date
4. Create the best representation of the jobseeker’s skills to date
5. Constantly help iterate for the user according to different jobs being applied for
Our conclusion is the CV Optimiser is the quickest route to market within the development lifecycle of the AIRE project and is now earning revenue.
We see a fantastic opportunity to develop the AIRE project in conjunction with this 1st phase product as the ability of our software is enhanced by extracting all the data received from 1,000’s of CV applications ensured our product is further embedded in the market with our machine learning getting real time data for dynamic capability.