Periodic Reporting for period 2 - HimL (Health in my Language)
Período documentado: 2016-08-01 hasta 2018-01-31
The goal of the HimL project was to increase timely access to important health information by making
it available to consumers in their own language, using machine translation (MT)
In particular we targetted two different use-cases:
1) Immigrant communities who have limited command of the official
language(s) of the country. They require information about the local health services, but it
is not available in their language. This use-case corresponds to NHS 24, who are responsible for providing public health service
information in Scotland.
2) Information about best practices in health care, resulting from recent research, are primarily
disseminated in English. Consumers would like to access new meta-analyses in their own language.
Our partner Cochrane is concerned with commissioning and distributing
health-related reviews, and would like to increase the reach of these revies using translation.
HimL aimed to develop MT systems to translate health-related texts into
Czech, German, Romanian and Polish, and to integrate these systems into the content management
systems of our user partners.
In HimL, we have pushed the boundaries on the state-of-the-art in MT,
and placed it into the healthcare domain, where accuracy is vitally important. Our conclusion was that using
MT can significantly reduce the cost of producing translated content for users like Cochrane
and NHS 24. There is still a difference in performance according to language, with translation into German
and Czech found to be more reliable than into Romanian, which is more reliable than
translation into Polish.
it available to consumers in their own language, using machine translation (MT)
In particular we targetted two different use-cases:
1) Immigrant communities who have limited command of the official
language(s) of the country. They require information about the local health services, but it
is not available in their language. This use-case corresponds to NHS 24, who are responsible for providing public health service
information in Scotland.
2) Information about best practices in health care, resulting from recent research, are primarily
disseminated in English. Consumers would like to access new meta-analyses in their own language.
Our partner Cochrane is concerned with commissioning and distributing
health-related reviews, and would like to increase the reach of these revies using translation.
HimL aimed to develop MT systems to translate health-related texts into
Czech, German, Romanian and Polish, and to integrate these systems into the content management
systems of our user partners.
In HimL, we have pushed the boundaries on the state-of-the-art in MT,
and placed it into the healthcare domain, where accuracy is vitally important. Our conclusion was that using
MT can significantly reduce the cost of producing translated content for users like Cochrane
and NHS 24. There is still a difference in performance according to language, with translation into German
and Czech found to be more reliable than into Romanian, which is more reliable than
translation into Polish.
The HimL workplan consisted of three annual cycles of research into quality improvements in MT, followed by deployment of improved systems and evaluation.
Each of these cycles included a release of a translation system for all 4 language pairs, which was then integrated into the user partners' content management systems.
We focused on three main areas of improvement for the MT:
1) Domain adaptation: Tuning the translation system for the public health domain.
2) Semantics: Improving the fidelity of the MT.
3) Morphology: Ensure correct morphological variants are produced.
When HimL began, we intended to use phrase-based or syntactic MT, since these were state-of-the-art
at the time. However during the first half of the project a new approach to MT, based on deep learning, was developed and
took over as the best performing technique. Since "neural MT" (NMT) was producing much better results, we decided to switch
the project to NMT, and use NMT for the final (Y3) release.
The switch to NMT entailed some adjustments to the workplan, but we were able to make improvements in the key areas on top
of the baseline NMT models. We deployed NMT within the same architecture as the earlier systems,
and integrated markup handling into the new paradigm.
User evaluation and user acceptance was an important part of HimL. We conducted extensive ranking experiments where users
compared our system and its variants, with a well-known online system, and our systems were able to offer better results in
all language pairs. In application-focused testing, Cochrane showed that post-editing using the HimL system (in the MateCat
tool) was 30-40% faster than translation from scratch for all languages except for Polish. As regards the use of raw (i.e.
unedited) MT on user websites, Cochrane noted large improvements within the life of the project, and traffic studies suggested
that using MT to add new languages (Czech and Romanian) could drive up traffic from those communities.
Each of these cycles included a release of a translation system for all 4 language pairs, which was then integrated into the user partners' content management systems.
We focused on three main areas of improvement for the MT:
1) Domain adaptation: Tuning the translation system for the public health domain.
2) Semantics: Improving the fidelity of the MT.
3) Morphology: Ensure correct morphological variants are produced.
When HimL began, we intended to use phrase-based or syntactic MT, since these were state-of-the-art
at the time. However during the first half of the project a new approach to MT, based on deep learning, was developed and
took over as the best performing technique. Since "neural MT" (NMT) was producing much better results, we decided to switch
the project to NMT, and use NMT for the final (Y3) release.
The switch to NMT entailed some adjustments to the workplan, but we were able to make improvements in the key areas on top
of the baseline NMT models. We deployed NMT within the same architecture as the earlier systems,
and integrated markup handling into the new paradigm.
User evaluation and user acceptance was an important part of HimL. We conducted extensive ranking experiments where users
compared our system and its variants, with a well-known online system, and our systems were able to offer better results in
all language pairs. In application-focused testing, Cochrane showed that post-editing using the HimL system (in the MateCat
tool) was 30-40% faster than translation from scratch for all languages except for Polish. As regards the use of raw (i.e.
unedited) MT on user websites, Cochrane noted large improvements within the life of the project, and traffic studies suggested
that using MT to add new languages (Czech and Romanian) could drive up traffic from those communities.
Within HimL, we have made progress beyond state-of-the-art in five areas, which we describe below.
Data and Adaptation: We have released a standard data set for training of medically-oriented
MT systems - the UFAL medical corpus (https://ufal.mff.cuni.cz/ufal_medical_corpus).
This covers 8 European language pairs, including the
HimL languages. Our main technique for adaptation has been fine-tuning the NMT systems
with in-domain data, where we extend the in-domain data with synthetic data created by selecting
from large mono-lingual crawls and back-translating. We also showed how to mine translations
from monolingual data, improving published techniques to work with medical terms and rare terms.
Semantics: Analysing the outputs of NMT shows that some problems that were apparent in earlier systems
re now extremely rare. However NMT still exhibits problems where
it omits important information, or adds incorrect information to the output. To counter this we use
a technique called "reconstruction", where the source should be reconstructable from the output.
We have also shown how to improve NMT using high quality dictionaries, and how to incorporate semantic
and syntactic information from external tools.
Morphology: NMT often produces more fluent, morphologically correct output, and indeed our analysis showed
that the placement and inflection of German verbs was much improved. However a consequence of the
move to NMT was that it was necessary to limit the vocabulary size using subwords. In HimL, we
developed a linguistically motivated way of segmenting German which improved over the standard, non-linguistic
segmentation method. We also showed how to improve NMT (in low-resource settings) by separately generating
the lemma and morphological tag of each word, and then synthesising.
To deploy our lab-based models, we wrapped our systems in a simple API which could be used by both NHS 24 and
Cochrane. The translation wrapper handles mark-up, and was re-engineered when we shifted to NMT in Year 3.
Our evaluation work has included ranking (generalised A-B tests), post-editing performance measurement and
user studies and surveys. We have also developed automatic and human assisted semantic metrics to
help us in tuning our systems towards fidelity. The user studies and surveys have looked at users' attitudes
towards MT and their assessment of its usefulness. We find that users want human-level
quality in translation (which is best achieved by post-editing) but they acknowledge that MT can be useful
if this is not possible.
In terms of scientfic impact, HimL has resulted in 55 scientific papers, listed on our website. We have
also contributed to the WMT17 biomedical shared translation task, making available a standard training set
as well as HimL test sets for all the HimL languages. HimL has contributed to the development
of two different open-source NMT systems (Nematus and HimL) which are gaining market-share amongst both
academic and non-academic users.
Outside the academic community, HimL can have an impact on the public health domain and beyond in other
public services. There are many cases where multilingual content is required and the improvements we have
seen in translation can make the production of this content (by post-editing) more economical and the
use of unedited MT (in cases where quality can be compromised) more feasible.
For Cochrane, the reviews and plain language summaries are a critical resource for healthcare consumers. A large
amount of Cochrane content has been translated into Spanish and French, and this has had a profound
effect on access to the Cochrane portal, indicating an unmet need. Cochrane do not have resources to reach all non-English speakers
with full human translation, so MT can help to bridge the gap.
Data and Adaptation: We have released a standard data set for training of medically-oriented
MT systems - the UFAL medical corpus (https://ufal.mff.cuni.cz/ufal_medical_corpus).
This covers 8 European language pairs, including the
HimL languages. Our main technique for adaptation has been fine-tuning the NMT systems
with in-domain data, where we extend the in-domain data with synthetic data created by selecting
from large mono-lingual crawls and back-translating. We also showed how to mine translations
from monolingual data, improving published techniques to work with medical terms and rare terms.
Semantics: Analysing the outputs of NMT shows that some problems that were apparent in earlier systems
re now extremely rare. However NMT still exhibits problems where
it omits important information, or adds incorrect information to the output. To counter this we use
a technique called "reconstruction", where the source should be reconstructable from the output.
We have also shown how to improve NMT using high quality dictionaries, and how to incorporate semantic
and syntactic information from external tools.
Morphology: NMT often produces more fluent, morphologically correct output, and indeed our analysis showed
that the placement and inflection of German verbs was much improved. However a consequence of the
move to NMT was that it was necessary to limit the vocabulary size using subwords. In HimL, we
developed a linguistically motivated way of segmenting German which improved over the standard, non-linguistic
segmentation method. We also showed how to improve NMT (in low-resource settings) by separately generating
the lemma and morphological tag of each word, and then synthesising.
To deploy our lab-based models, we wrapped our systems in a simple API which could be used by both NHS 24 and
Cochrane. The translation wrapper handles mark-up, and was re-engineered when we shifted to NMT in Year 3.
Our evaluation work has included ranking (generalised A-B tests), post-editing performance measurement and
user studies and surveys. We have also developed automatic and human assisted semantic metrics to
help us in tuning our systems towards fidelity. The user studies and surveys have looked at users' attitudes
towards MT and their assessment of its usefulness. We find that users want human-level
quality in translation (which is best achieved by post-editing) but they acknowledge that MT can be useful
if this is not possible.
In terms of scientfic impact, HimL has resulted in 55 scientific papers, listed on our website. We have
also contributed to the WMT17 biomedical shared translation task, making available a standard training set
as well as HimL test sets for all the HimL languages. HimL has contributed to the development
of two different open-source NMT systems (Nematus and HimL) which are gaining market-share amongst both
academic and non-academic users.
Outside the academic community, HimL can have an impact on the public health domain and beyond in other
public services. There are many cases where multilingual content is required and the improvements we have
seen in translation can make the production of this content (by post-editing) more economical and the
use of unedited MT (in cases where quality can be compromised) more feasible.
For Cochrane, the reviews and plain language summaries are a critical resource for healthcare consumers. A large
amount of Cochrane content has been translated into Spanish and French, and this has had a profound
effect on access to the Cochrane portal, indicating an unmet need. Cochrane do not have resources to reach all non-English speakers
with full human translation, so MT can help to bridge the gap.