Get the app
Dermatology2018Impact: 2 out of 3, validated at scale

Melanoma: an AI put to the test against 58 dermatologists

A study published in Annals of Oncology in 2018 compared a neural network with 58 dermatologists from several countries at recognising melanoma on dermoscopy images (photos of the skin taken through the dermatologist's special magnifier). Most doctors were outperformed by the AI, which had a higher specificity: 82.5% against 71.3%.

Annals of Oncology, 2018Lire en français

Why compare an AI with a large group of doctors?

Neural networks, programs that learn to recognise images, could help detect melanoma, the most serious skin cancer. But earlier studies mostly compared the AI with small groups of doctors. Data against larger groups were lacking.

This study set out to measure the gap against a large international panel, to learn what the tool really adds compared with how dermatologists actually perform.

How was the experiment run?

The researchers used a Google neural network (Inception v4), trained and validated on dermoscopy images with their matching diagnoses. They then gave a 100-image test to 58 dermatologists from several countries, including 30 experts.

The test had two levels. At level I, doctors saw only the dermoscopy image. At level II, they also received clinical information and images. Each time, they had to say whether the lesion was malignant or not.

The study is registered in the German Clinical Trial Register (DRKS00013570), which means the method was announced in advance.

What do the results show?

At level I, dermatologists spotted on average 86.6% of melanomas (sensitivity) and correctly cleared 71.3% of harmless lesions (specificity). With clinical information added, sensitivity rose to 88.9% and specificity to 75.7%.

When the AI is set to spot as many melanomas as the doctors do, it clears harmless lesions more often: 82.5% specificity. The area under the ROC curve (an overall score where 1 is perfect) was 0.86 for the AI against 0.79 on average for dermatologists.

The program was also compared with the top five algorithms of a 2016 international challenge (ISBI) and scored close to the top three. Adding clinical information improved the doctors' performance, which shows how much context matters.

Key figures

  • 58dermatologists from several countries compared with the AI, including 30 experts
  • 82.5%specificity for the AI, against 71.3% for dermatologists (level I)
  • 0.86overall AI score (area under the ROC curve), against 0.79 on average for doctors

What does this change for patients?

For a patient, better specificity means in theory fewer harmless lesions mistaken for cancer, so fewer unnecessary removals, without missing more melanomas. The authors conclude that doctors, whatever their experience, could benefit from an AI's help in classifying images.

This idea of helping the doctor is the most realistic one: it is a second opinion, not a replacement.

The authors specify that this benefit holds whatever the doctor's experience: beginners and experts alike could gain. How to fit this opinion into daily work remains to be settled.

What are the limits of this study?

The test used 100 images: it is an experimental setting, much simpler than a consultation where the doctor examines the whole skin and knows the patient's history. The doctors also did not work in their usual conditions.

The study does not say whether patients are better cared for in practice with this tool. Trials in real conditions are still needed.

A mole that worries you: preparing the consultation

In this study the AI did as well as many dermatologists on images, with fewer unnecessary removals. That is not yet happening in every clinic, and your dermatologist makes the decision.

With Mima, you note how long a spot has bothered you and what has changed, then prepare your question: "Does this mole need monitoring or removal?" She keeps track of your appointments and the explanations you were given.

Frequently asked questions

Is AI better than dermatologists at melanoma?

In this 100-image test, it outperformed most of the 58 dermatologists on specificity. It was an experimental setting: it does not mean it does better than a doctor in a consultation.

What is specificity?

It is the share of harmless lesions that the test correctly clears. The higher it is, the less often a harmless lesion is mistaken for cancer.

What is dermoscopy?

It is a skin examination with a special lit magnifier, the dermatoscope, which shows details invisible to the naked eye.

Is this tool used in clinics?

The study does not say so. It compares the AI with doctors on a set of images and concludes that they could benefit from its help.

The original study

Haenssle HA, Fink C, Schneiderbauer R, et al. “Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.” Annals of Oncology, August 2018. DOI : 10.1093/annonc/mdy166

Read the full study in Annals of Oncology

This article summarises a scientific study and is not medical advice.