Using a Few Expert Labels to Make Crowd Annotations More Reliable

Crowdsourcing can spread the work of labeling data across many people, but it creates another problem: deciding whose answers deserve weight. Workers may understand the task well, guess, or consistently choose the wrong answer. If unreliable workers become numerous enough, an aggregation method can suffer a sharp collapse in accuracy.

In their preprint, Faiza Khan Khattak and Ansaf Salleb-Aouissi describe Expert Label Injected Crowd Estimation, or ELICE. The framework uses expert judgments on a small subset of items to assess crowd workers, estimate item difficulty and combine their answers. The supplied evidence does not establish peer-reviewed publication of this manuscript.

A small reference sample

ELICE starts with items labeled by both experts and crowd workers. Agreement with the expert answers provides an estimate of each worker’s ability. The proportion of workers answering an item correctly also helps estimate its difficulty. For items without expert labels, the method first constructs provisional answers from the crowd and uses those to estimate difficulty.

The simplest version, ELICE 1, combines answers through a weighted vote. A logistic function converts worker-ability and item-difficulty estimates into weights. The first two versions assume the expert answers are correct, making the quality and representativeness of the reference sample important.

Cluster-based variants select equal numbers of reference items from groups formed using item features, rather than sampling only from the dataset as a whole. This is intended to broaden coverage, particularly for uneven datasets, but clusters need not correspond exactly to the answer classes.

When wrong answers carry information

ELICE 2 distinguishes near-chance performance from consistently correct or consistently incorrect performance. It uses an entropy-based score: uncertainty is greatest when a worker’s accuracy is near one-half and lower toward either extreme.

For a binary task, a reliably wrong answer can be reversed. ELICE 2 therefore flips answers from workers assessed as performing below chance and then weights their contributions. Random answers, by contrast, offer little dependable information to recover.

The authors call consistently incorrect workers “malicious,” including people who misunderstand the task. Their performance score does not establish anyone’s intentions. The label-flipping mechanism also relies on the binary setting: with several possible answers, knowing that one answer is wrong does not identify the correct alternative.

ELICE 3 addresses the possibility that experts themselves are mistaken or disagree. Starting from ELICE 2 estimates, it adds model-based comparisons between workers and between items. A circular variant compares neighboring members in a cycle rather than every possible pair, reducing the number of comparisons. These are computational comparisons, not additional judgments collected from workers.

Strong simulations, mixed task-level results

The authors tested five UCI datasets with simulated crowd answers, generating 20 labels per item by flipping existing reference labels. Simulated good workers made fewer than 35% errors, random workers made roughly 35–65%, and malicious workers made roughly 65–100%.

In the table’s category with more than 70% random and/or malicious workers, ELICE 2 achieved approximately 97.8–99.3% accuracy across the five datasets, averaged over 50 runs. Majority voting ranged from about 3.0% to 10.6%. These are striking results under the specified simulation, not measurements of how arbitrary real crowds behave. When unreliable workers were scarce, several comparison methods equaled or exceeded ELICE variants, so the results do not support universal superiority.

The real-crowd experiments used Amazon Mechanical Turk. For 100 mammograms, workers distinguished malignant images from other categories. With eight reference items, ELICE Circular achieved 77.22% accuracy, compared with 55% for majority voting; the reported results average 100 runs. This was an annotation experiment, not evidence of clinical diagnostic readiness.

Three additional experiments asked workers to assign photographs to the paper’s specified racial categories. Results varied: ELICE 2 performed best on one comparison, while ELICE 1 performed best on another and the later versions did worse. These results concern agreement with the dataset’s assigned categories, not an independently established ability to determine identity from appearance.

Limits to the promise

Full pairwise comparison was computationally expensive: the authors could not report it for the 8,124-item Mushroom dataset under their hardware constraints. The paper also proposes a lower bound on expert-label requirements using a modeled error function and a PAC-learning-style argument. That model-dependent calculation should not be read as a universally validated annotation-budget guarantee.

ELICE’s central contribution is a way to use limited expert input to interpret crowd answers rather than simply count them. Its effectiveness still depends on the reference labels, the task and how well the assumed error patterns match actual workers.