Periodic Reporting for period 1 - IRSkin (IRSkin provides dermatologists a new dermatoscope, equipped with a software for visual comparisons in the diagnostic phase, performing a content based Image Retrieval from an a-priori built database.)
Periodo di rendicontazione: 2015-07-01 al 2015-10-31
IRSkin aims to provide to the dermatologist and to the dermoscopy expert clinician a visual support in the diagnosis phase, based on a thoroughly classified Image Retrieval System (IRS) and on a Persistent Homology algorithm (Figure "Retrieval.JPG" shows an IR example and Figure "Retrieval2.JPG" shows the scheme of the Persistent Homology-based Retrieval algorithm). The key objective of the Phase 1 project is the elaboration of a detailed Business Plan achievable within the duration of the project, which includes activities related to detailed measurable specific objectives listed below:
Objective 1:
Customer validation and assessment of the clinical and technical potential of the IRSkin dermatoscope (see Figure "Dermatoscope.JPG"), by interviewing international dermatologists, with the aim to:
- better understand the dermatologists’ needs;
- assess the existing dermatoscopes and the satisfaction of their users;
- assess the IRSkin added value on existing solutions;
- get a qualified opinion on the clinical and technical features of IRSkin;
- understand if dermatologists are interested in buying the IRSkin video dermatoscope or the reasons that would prevent them from using it;
- evaluate how much a dermatologist would pay to buy the IRSkin dermatoscope;
- identify possible differences between the customer’s types, through an a-priori assessment of the dermatologists’ experience.
Objective 2:
Assessment of the legal (contractual) and ethical framework for implementing a database of diagnosed images, to be used for testing the algorithm and in the preventive diagnostic phase by the IRSkin tool. Assessment of how IRSkin meets the national legal and ethical requirements of the countries to which it is addressed.
Objective 3:
Market analysis to assess the existing and potential market of IRSkin:
- potential market of dermatology devices (volume and revenues);
- incidence, mortality and 5-years prevalence rate of melanoma in Europe and in the World;
- incidence and mortality growth forecast of melanoma in 2020 and 2025;
- number of dermatologists with respect to the population in different countries;
- personal cost for a biopsy and total cost for the health care of melanoma in Europe.
Objective 4:
Clinical study to estimate the improvement of diagnosis accuracy and quantify the consequent economic benefits.
Objective 5:
Consolidate all the information collected into a detailed Business Plan.
Objective 1:
Customer validation and assessment of the clinical and technical potential of the IRSkin dermatoscope (see Figure "Dermatoscope.JPG"), by interviewing international dermatologists, with the aim to:
- better understand the dermatologists’ needs;
- assess the existing dermatoscopes and the satisfaction of their users;
- assess the IRSkin added value on existing solutions;
- get a qualified opinion on the clinical and technical features of IRSkin;
- understand if dermatologists are interested in buying the IRSkin video dermatoscope or the reasons that would prevent them from using it;
- evaluate how much a dermatologist would pay to buy the IRSkin dermatoscope;
- identify possible differences between the customer’s types, through an a-priori assessment of the dermatologists’ experience.
Objective 2:
Assessment of the legal (contractual) and ethical framework for implementing a database of diagnosed images, to be used for testing the algorithm and in the preventive diagnostic phase by the IRSkin tool. Assessment of how IRSkin meets the national legal and ethical requirements of the countries to which it is addressed.
Objective 3:
Market analysis to assess the existing and potential market of IRSkin:
- potential market of dermatology devices (volume and revenues);
- incidence, mortality and 5-years prevalence rate of melanoma in Europe and in the World;
- incidence and mortality growth forecast of melanoma in 2020 and 2025;
- number of dermatologists with respect to the population in different countries;
- personal cost for a biopsy and total cost for the health care of melanoma in Europe.
Objective 4:
Clinical study to estimate the improvement of diagnosis accuracy and quantify the consequent economic benefits.
Objective 5:
Consolidate all the information collected into a detailed Business Plan.
Objective 1
CA-MI S.r.l. developed a questionnaire divided into two parts: one dedicated to the users’ needs (current solutions and users’ satisfaction), the other to the IRSkin’s video dermatoscope as a new solution.
The questionnaire is available at the following link (see also the URL of the action's public website):
http://goo.gl/forms/ygX1YCpoEd
The questionnaire was submitted during October (1-month period) to a large number of international dermatologists, in order to assess the clinical and technical potential of the IRSkin's video dermatoscope and for the customer validation:
- via e-mail to 3,529 international dermatologists (by buying the mailing list by MedList International):
14 from Africa;
39 from Australia and New Zealand;
623 from Asia and Japan;
65 from Europe;
1510 from Western Europe;
89 from Middle East;
77 from South America;
181 from UK;
931 from USA and Canada;
- by interviewing international dermatologists:
in their personal office (Forlì, Torino, Padova, etc.);
at the following conferences: Copenhagen (7-11 October) and Salsomaggiore Terme (22-24 October).
All dermatologists were requested to provide data in relation to their personal information, their experience in the dermatology field, their needs, the current solutions on the market and their interest in the IRSkin dermatoscope. It took about 15-20 minutes to each dermatologist to see the presentation video of IRSkin and to complete the questionnaire.
Main results:
1) Needs: 88% of dermatologists declared as first need the high-definition acquisition of skin lesion images, in combination to other answers. Dermatologists expressed a higher interest in Retrieval function rather than in Risk classification. 51.2% of them answered Retrieval function whereas 23.3% answered Risk classification, in combination to other answers (see Figure "Needs.JPG").
2) Interest in the project: 88% of dermatologists declared that IRSkin meets at least partially their needs (see Figure "IRSkin.JPG").
3) 89% of respondents would use for sure IRSkin if it were free and 30% declare to be willing to buy it even at a cost of 25k € (see Figure "IRSkin2.JPG").
The detailed results are described in the Deliverable D1.1 of Phase 1.
Objective 2
For implementing a database of diagnosed images to be used for testing the algorithm and in the preventive diagnostic phase by the IRSkin video dermatoscope, IRST and CA-MI prepared an ethical protocol (containing the research methodology, inclusion criteria, exclusion criteria, the data collection process, clinicians involved, appropriate informed consent by the patient etc.) formalized by the IRST’s Medical and Scientific Committee and the IRST’s Ethical Committee. Specifically, informed Consent will be subscribed by the patient prior to any procedure and will include full disclosure of how images are to be taken, stored, and de-identified.
Any other Research Centre that will be involved in the future implementation of the IRSkin Data Set will have to prepare an ethical protocol, according to the National requirements, before starting the research activities. The diagnosed skin images are property of Research Centres and CA-MI is authorised to use them for clinical motivations.
When IRSkin will be CE certified, it will meet the national legal and ethical requirements of all European countries and no integration will be required.
Objective 3
CA-MI conducted a Market analysis (see Figure "Competitors.JPG"), contacting International Associations of Dermatology and consulting articles and reports (for example on ReportLinker), to assess the existing and potential market of IRSkin. Melanoma is a very common cancer all over the World (especially in fair skinned Caucasian population) and its incidence is growing, as well as the costs associated.
The market is a niche (1 dermatologist per about 50,000 people) but with good revenues ($6.578,23 millions in 2014) and growth perspectives (+11.50 % of growth in revenues estimated in 2019).
Objective 4
The IRSkin’s algorithm was tested twice on a dataset of 107 diagnosed images of IRST’s property, containing 35 melanomas and 72 nevi, acquired in epiluminescence microscopy with a fixed 16-fold magnification. The only selection criteria were that the lesion had to be entirely visible and that the resolution had to be fixed at 768 × 576 pixels.
This preliminary test consisted in the "improper" use of the Retrieval System as a classifier, following the k-Nearest Neighbour paradigm, because of the easy comparison of the output with an objective ground truth.
The test was conducted by optimizing only five 1-dimensional Persistence Diagrams, or Persistent Betti Numbers (PBNs) and obtained the following results:
- Sensitivity: 88,6%
- Specificity: 91,7%
In a second test, some new 1-dimensional PBNs and a recent algorithm for 2-dimensional PBNs1 were implemented to the aim of improving the performances of the software. The results obtained by the optimization of the PBNs features separately were compared to those achieved by a retrieval based on 10 ABCDE features, getting the following remarkable result (see also Figure "PHvsABCD.JPG"):
Features Accuracy (%) Sensitivity (%) Specificity (%)
1d PBNs 94.39 94.29 94.44
2d PBNs 92.52 88.57 94.44
ABCD(E) 86.92 82.86 88.89
Accuracy refers to the number of lesions diagnosed correctly on the total of lesions, Sensitivity refers to the number of melanomas diagnosed correctly on the total of melanomas and Specificity refers to the number of nevi diagnosed correctly on the total of nevi.
The table shows that the classification systems based on the PBNs were able to distinguish nevi and melanomas in a very precise way, achieving better results than the ABCD(E) features. In particular, the results obtained by the 1-dimensional PBNs functions show that only a few of the 107 images were classified in a wrong way by the retrieval based on these functions. However, at least in this test, the 2-dimensional PBNs failed to provide an improvement in the results despite their demanding computational effort.
This remark was the starting point to consider new possibilities, in which the ABCD features are used at an early stage to discriminate between very different images and then the 1-dimensional PBNs come into play.
Although this test showed that the software is likely to compete with other self-diagnosis existing algorithms, the philosophy of the project goes far beyond a risk analysis or a sensitivity and specificity computation achieved through an automatic classification.
The final goal is that the dermatologist can use the device to quickly retrieve quantitative and qualitative information about a suspected lesion and thus improve his / her diagnosis skills.
For this reason, a clinical test was started to evaluate the effective visual results obtained by the algorithm. Thus, for each image in a certain randomly chosen subset of a large database, the retrievals computed by the software were evaluated by the experts of IRST from a clinical point of view.
The test was conducted on a dataset of 295 diagnosed images of IRST’s property, containing 67 melanomas and 228 nevi. From this database 30 queries were extracted and for each of them three different retrieval with 10 neighbours were performed. The first retrieval was based only on PBN functions, while the other two used a first discrimination based on some ABCD parameters and then the PBN functions to actually determine the neighbours. For every image retrieved, three clinicians rated the comparison with its query, based on three parameters (board, colour and texture), providing an overall rating from 0 to 3 points.
The results of this test mainly confirmed that the automatic visual retrieval may be a more useful support than an automatic classification of the suspected image. The full results of this test are described in the Deliverable D1.1 of Phase 1.
From the results of these tests, the clinicians evaluated a possible spare of 5% of unnecessary biopsies in their own specialized clinic.
Currently accuracy in melanoma detection has improved only in specialized clinics. Here, among removed skin lesions, the ratio between melanomas and nevi is 1/8, whereas in a general dermatological department it is 1/29 (Argenziano et al JAAD 2011). In this clinical context, the IRSkin tool is expected to increase the dermatologists’ level of expertise, improving the early diagnosis of melanoma and decreasing unneeded biopsies from 5 to 20%.
This aspect would be a significant gain for the public healthcare system or for private clinics: the cost of a biopsy is about 250-300 euro including the costs of the doctor, the nurse, the material of the operating room, histology and check-ups. Moreover, the opportunity to learn from a catalogue of already diagnosed images allows dermatologists to have a continuing education at no additional costs, which is especially useful for young dermatologists.
Objective 5
All results described above are collected in detail in the Deliverable D1.1 of Phase 1. A graphical summary of the financial plan is shown in Figure "FinancialPlan.JPG".
CA-MI S.r.l. developed a questionnaire divided into two parts: one dedicated to the users’ needs (current solutions and users’ satisfaction), the other to the IRSkin’s video dermatoscope as a new solution.
The questionnaire is available at the following link (see also the URL of the action's public website):
http://goo.gl/forms/ygX1YCpoEd
The questionnaire was submitted during October (1-month period) to a large number of international dermatologists, in order to assess the clinical and technical potential of the IRSkin's video dermatoscope and for the customer validation:
- via e-mail to 3,529 international dermatologists (by buying the mailing list by MedList International):
14 from Africa;
39 from Australia and New Zealand;
623 from Asia and Japan;
65 from Europe;
1510 from Western Europe;
89 from Middle East;
77 from South America;
181 from UK;
931 from USA and Canada;
- by interviewing international dermatologists:
in their personal office (Forlì, Torino, Padova, etc.);
at the following conferences: Copenhagen (7-11 October) and Salsomaggiore Terme (22-24 October).
All dermatologists were requested to provide data in relation to their personal information, their experience in the dermatology field, their needs, the current solutions on the market and their interest in the IRSkin dermatoscope. It took about 15-20 minutes to each dermatologist to see the presentation video of IRSkin and to complete the questionnaire.
Main results:
1) Needs: 88% of dermatologists declared as first need the high-definition acquisition of skin lesion images, in combination to other answers. Dermatologists expressed a higher interest in Retrieval function rather than in Risk classification. 51.2% of them answered Retrieval function whereas 23.3% answered Risk classification, in combination to other answers (see Figure "Needs.JPG").
2) Interest in the project: 88% of dermatologists declared that IRSkin meets at least partially their needs (see Figure "IRSkin.JPG").
3) 89% of respondents would use for sure IRSkin if it were free and 30% declare to be willing to buy it even at a cost of 25k € (see Figure "IRSkin2.JPG").
The detailed results are described in the Deliverable D1.1 of Phase 1.
Objective 2
For implementing a database of diagnosed images to be used for testing the algorithm and in the preventive diagnostic phase by the IRSkin video dermatoscope, IRST and CA-MI prepared an ethical protocol (containing the research methodology, inclusion criteria, exclusion criteria, the data collection process, clinicians involved, appropriate informed consent by the patient etc.) formalized by the IRST’s Medical and Scientific Committee and the IRST’s Ethical Committee. Specifically, informed Consent will be subscribed by the patient prior to any procedure and will include full disclosure of how images are to be taken, stored, and de-identified.
Any other Research Centre that will be involved in the future implementation of the IRSkin Data Set will have to prepare an ethical protocol, according to the National requirements, before starting the research activities. The diagnosed skin images are property of Research Centres and CA-MI is authorised to use them for clinical motivations.
When IRSkin will be CE certified, it will meet the national legal and ethical requirements of all European countries and no integration will be required.
Objective 3
CA-MI conducted a Market analysis (see Figure "Competitors.JPG"), contacting International Associations of Dermatology and consulting articles and reports (for example on ReportLinker), to assess the existing and potential market of IRSkin. Melanoma is a very common cancer all over the World (especially in fair skinned Caucasian population) and its incidence is growing, as well as the costs associated.
The market is a niche (1 dermatologist per about 50,000 people) but with good revenues ($6.578,23 millions in 2014) and growth perspectives (+11.50 % of growth in revenues estimated in 2019).
Objective 4
The IRSkin’s algorithm was tested twice on a dataset of 107 diagnosed images of IRST’s property, containing 35 melanomas and 72 nevi, acquired in epiluminescence microscopy with a fixed 16-fold magnification. The only selection criteria were that the lesion had to be entirely visible and that the resolution had to be fixed at 768 × 576 pixels.
This preliminary test consisted in the "improper" use of the Retrieval System as a classifier, following the k-Nearest Neighbour paradigm, because of the easy comparison of the output with an objective ground truth.
The test was conducted by optimizing only five 1-dimensional Persistence Diagrams, or Persistent Betti Numbers (PBNs) and obtained the following results:
- Sensitivity: 88,6%
- Specificity: 91,7%
In a second test, some new 1-dimensional PBNs and a recent algorithm for 2-dimensional PBNs1 were implemented to the aim of improving the performances of the software. The results obtained by the optimization of the PBNs features separately were compared to those achieved by a retrieval based on 10 ABCDE features, getting the following remarkable result (see also Figure "PHvsABCD.JPG"):
Features Accuracy (%) Sensitivity (%) Specificity (%)
1d PBNs 94.39 94.29 94.44
2d PBNs 92.52 88.57 94.44
ABCD(E) 86.92 82.86 88.89
Accuracy refers to the number of lesions diagnosed correctly on the total of lesions, Sensitivity refers to the number of melanomas diagnosed correctly on the total of melanomas and Specificity refers to the number of nevi diagnosed correctly on the total of nevi.
The table shows that the classification systems based on the PBNs were able to distinguish nevi and melanomas in a very precise way, achieving better results than the ABCD(E) features. In particular, the results obtained by the 1-dimensional PBNs functions show that only a few of the 107 images were classified in a wrong way by the retrieval based on these functions. However, at least in this test, the 2-dimensional PBNs failed to provide an improvement in the results despite their demanding computational effort.
This remark was the starting point to consider new possibilities, in which the ABCD features are used at an early stage to discriminate between very different images and then the 1-dimensional PBNs come into play.
Although this test showed that the software is likely to compete with other self-diagnosis existing algorithms, the philosophy of the project goes far beyond a risk analysis or a sensitivity and specificity computation achieved through an automatic classification.
The final goal is that the dermatologist can use the device to quickly retrieve quantitative and qualitative information about a suspected lesion and thus improve his / her diagnosis skills.
For this reason, a clinical test was started to evaluate the effective visual results obtained by the algorithm. Thus, for each image in a certain randomly chosen subset of a large database, the retrievals computed by the software were evaluated by the experts of IRST from a clinical point of view.
The test was conducted on a dataset of 295 diagnosed images of IRST’s property, containing 67 melanomas and 228 nevi. From this database 30 queries were extracted and for each of them three different retrieval with 10 neighbours were performed. The first retrieval was based only on PBN functions, while the other two used a first discrimination based on some ABCD parameters and then the PBN functions to actually determine the neighbours. For every image retrieved, three clinicians rated the comparison with its query, based on three parameters (board, colour and texture), providing an overall rating from 0 to 3 points.
The results of this test mainly confirmed that the automatic visual retrieval may be a more useful support than an automatic classification of the suspected image. The full results of this test are described in the Deliverable D1.1 of Phase 1.
From the results of these tests, the clinicians evaluated a possible spare of 5% of unnecessary biopsies in their own specialized clinic.
Currently accuracy in melanoma detection has improved only in specialized clinics. Here, among removed skin lesions, the ratio between melanomas and nevi is 1/8, whereas in a general dermatological department it is 1/29 (Argenziano et al JAAD 2011). In this clinical context, the IRSkin tool is expected to increase the dermatologists’ level of expertise, improving the early diagnosis of melanoma and decreasing unneeded biopsies from 5 to 20%.
This aspect would be a significant gain for the public healthcare system or for private clinics: the cost of a biopsy is about 250-300 euro including the costs of the doctor, the nurse, the material of the operating room, histology and check-ups. Moreover, the opportunity to learn from a catalogue of already diagnosed images allows dermatologists to have a continuing education at no additional costs, which is especially useful for young dermatologists.
Objective 5
All results described above are collected in detail in the Deliverable D1.1 of Phase 1. A graphical summary of the financial plan is shown in Figure "FinancialPlan.JPG".
Economic benefits:
- Reduction of health care system costs;
- Sustainability of the health care system;
- Reduction of learning costs to train new dermatologists;
- Reduction of costs associated to a biopsy.
Learning benefits:
- A high number of images with a correct diagnosis available for training sessions;
- Improving of the learning curve of dermatologists in short term using interactive IRS;
- Automatic and reliable feedback on the dermatologists’ perception of the risk of melanoma.
Clinical benefits:
- Increase of the diagnosis accuracy;
- Reduction of the waiting time of the diagnosis;
- Improvement of clinical decisions;
- Reduction of unneeded and invasive biopsies.
- Reduction of health care system costs;
- Sustainability of the health care system;
- Reduction of learning costs to train new dermatologists;
- Reduction of costs associated to a biopsy.
Learning benefits:
- A high number of images with a correct diagnosis available for training sessions;
- Improving of the learning curve of dermatologists in short term using interactive IRS;
- Automatic and reliable feedback on the dermatologists’ perception of the risk of melanoma.
Clinical benefits:
- Increase of the diagnosis accuracy;
- Reduction of the waiting time of the diagnosis;
- Improvement of clinical decisions;
- Reduction of unneeded and invasive biopsies.