Examples improve Roman Urdu toxicity detection, but prompt-tuning gains remain modest

Roman Urdu writes Urdu in the Latin alphabet, often with inconsistent spellings, phonetic variations and informal grammar. Social media comments also mix languages and depend on context. These features make it difficult for a classifier to distinguish harmful language from ordinary conversation.

A preprint by Toneema Zubair and colleagues investigates two ways of adapting language models to this setting: changing what a model sees in its prompt, and training small prompt representations. Its detailed results support a substantial improvement from giving GPT-3.5 labeled examples, but a considerably smaller improvement from mixed prompt tuning with multilingual BERT.

Two different kinds of adaptation

The study uses PURUTT, a dataset containing 72,771 social media comments: 13,097 labeled toxic and 59,674 non-toxic. Although the paper describes its application as hate speech detection, the actual classification task is broader. Its instructions include offensive language and other harmful content within the toxic category.

For prompt engineering, the researchers accessed GPT-3.5 through Azure OpenAI. In the zero-shot condition, the model received instructions and a comment, then returned a binary label. In the few-shot condition, the prompt also contained five toxic and five non-toxic examples. Neither approach updated the model’s parameters.

Prompt tuning used a different model and procedure. With multilingual BERT and the OpenPrompt framework, classification became a masked-word prediction task. A manually chosen set of label words connected predictions to toxic or non-toxic classes.

The researchers compared two fixed template arrangements, called prefix and cloze prompts, with a mixed approach incorporating trainable soft-token embeddings. The mixed method is described as keeping the underlying language model frozen while optimizing these prompt embeddings. The manual prefix template should not be confused with learning a continuous prefix: here, it is a fixed textual arrangement.

Examples produced the clearest reported improvement

The detailed GPT-3.5 results give few-shot prompting an F1 score of 0.8565, compared with 0.6698 for zero-shot prompting. Reported accuracy rose from 0.7229 to 0.9100. F1 combines precision and recall, making it useful alongside accuracy when one class is much less common than the other.

These figures require attention to how they were calculated. The paper says comments were processed in batches of 100 and that final metrics were averages of batch-level metrics. They should therefore not be read as necessarily equivalent to scores calculated once over all pooled predictions. The maximum F1 values reported separately describe the best observed batches, not overall performance.

For BERT, the strongest mixed-prompt result was an F1 of 0.6852 at K = 128, compared with 0.6813 for the best prefix configuration. The best cloze result was 0.6532 at K = 32. Mixed prompting’s advantage over the best fixed prefix was therefore small in the reported table. Prefix prompting also achieved higher accuracy: 0.8048, versus 0.7498 for the mixed configuration with the best F1.

Conflicting descriptions limit the conclusions

The preprint contains material inconsistencies. Its abstract describes mixed prompting as reaching about 0.87 F1, whereas the detailed tables and conclusion give 0.6852. The abstract also highlights a Mistral prompting comparison, while the main prompt-engineering experiment concerns GPT-3.5. The detailed results, rather than those abstract claims, underpin the findings described here.

The experimental setup is also described differently across sections. One account specifies a 60/20/20 dataset split; another assigns K examples each to training and development and the remainder to testing. Training duration is variously given as two or three epochs. These discrepancies leave the precise evaluation procedure uncertain.

For GPT-3.5, the algorithm selects demonstration examples from the batch being processed without clearly explaining whether those examples are excluded from scoring. This creates uncertainty about evaluation separation, rather than establishing that every scored comment was unseen within its prompt.

Previously published PEFT results, including a Mistral F1 of 0.9387, provide context, not a newly controlled head-to-head experiment. The authors acknowledge that experimental differences limit direct comparison. Cross-domain testing and evaluation in additional languages remain future work.

The practical finding is consequently narrower than a general victory for prompting: labeled examples improved GPT-3.5’s reported results on this dataset, while mixed BERT prompts only narrowly exceeded the best fixed template. Resolving the reporting inconsistencies is important before drawing stronger conclusions about reliability or deployment.