A structured benchmark for AI and everyday skin photos

Why build a different dermatology benchmark?

Many dermatology image benchmarks use dermatoscopic images: close-up views made with a specialized device. These datasets have helped advance image analysis, but they do not necessarily reflect how people ask for help—by sharing ordinary photos alongside descriptions of what they have noticed.

DermaVQA-DAS extends an existing dataset of patient-generated dermatology images and queries. It adds two tasks: answering structured, multiple-choice questions about a skin concern, and marking a lesion’s location in an image. The researchers aim to make performance easier to compare while testing models with patient-authored context, not images alone.

At the center is the Dermatology Assessment Schema, or DAS. It organizes observations such as affected area, size, color, texture and itch into questions with predefined answer choices. The paper’s methods and main text describe 72 high-level questions and 137 fine-grained questions, including repeated fields for multiple affected sites. Questions and choices are available in English and Chinese. The reported experiments use a subset: nine categories covering 27 fine-grained questions, selected because they were populated for at least 51% of encounters in the test and validation splits.

How the tests worked

For question answering, a model received an image, the patient’s original query and a DAS assessment question. Its response had to use the schema’s answer options. The QA test set contained 100 instances; the full dataset also included 300 training and 56 validation instances. For cases involving multiple images or locations, researchers used specified rules to combine predictions. Scoring allowed partial credit for partial matches—for example, naming one of two annotated locations.

For segmentation, the researchers evaluated MedSAM and BiomedParse using different prompting setups. MedSAM was tested without a text prompt; BiomedParse was tested with three prompts, ranging from a default instruction to prompts that also included the patient query title and, in one condition, its full content. The segmentation collection comprised 998 encounters, 2,945 images and 8,864 expert-annotated masks.

Because multiple annotators supplied masks, the study reported several ways of aggregating results. Mean-of-Max gives each example its best overlap among reference masks, while majority-vote microscore compares against a pixel-level consensus mask. These approaches measure overlap differently, so their scores are not directly interchangeable.

What the models did—and did not—show

Across evaluated systems, overall closed-ended QA accuracy ranged from 0.729 to 0.798. The highest overall score was 0.798 for o3, closely followed by GPT-4.1 at 0.796. Gemini-1.5-Pro scored 0.783. These are results on this benchmark, not measures of diagnostic accuracy in clinical practice.

Results varied by question type. Itch questions, which can draw on patient-reported information, were comparatively strong. Color and texture questions showed greater variation across models, which the authors interpret as potentially reflecting greater difficulty or ambiguity in recognizing fine visual attributes and selecting from a controlled vocabulary.

In segmentation, BiomedParse generally scored above MedSAM. For BiomedParse, the prompt including both the patient query title and content produced the highest scores under majority-vote evaluation: Jaccard 0.3948 and Dice 0.5661. Under the other reported aggregation methods, the default prompt performed best. So the strongest prompt depended on the scoring approach.

The authors describe segmentation performance as promising but modest. They point to variation in subject focus, angle and lighting in consumer-supplied photos as possible challenges; their limitations also mention image resolution. The results show that structured questions differ in difficulty, but they do not establish that every fine-grained visual judgment is difficult.

Limits and next steps

This is a benchmark study, not a clinical trial. Scores on a fixed set of images and questions do not establish that a system can diagnose a new patient, improve care or work safely without clinician oversight. The authors say the dataset’s size and clinical diversity remain limited relative to real-world dermatology practice, potentially restricting generalization to rare conditions, different skin tones and varied imaging settings. A fixed menu of answer choices may also miss patient concerns or clinical reasoning that do not fit the schema.

The paper says the dataset, schema and evaluation protocols are publicly released for research. The supplied manuscript is an arXiv preprint, which may not have undergone peer review. Its abstract and main text also give inconsistent DAS question counts: the abstract in the supplied source record reports 36 high-level and 27 fine-grained questions, while the paper’s methods and main text report 72 and 137. The figures above follow the detailed methods and main text; readers should consult the latest source version when using these counts.

For now, DermaVQA-DAS is best understood as a tool for measuring specific capabilities—not proof that AI can reliably assess skin problems from patient photos. Its contribution is to make some of those capabilities, and their limits, more explicit and testable.