AI panel ranks parties’ green pledges—but reads them as greener than experts do

Reading pledges, not measuring delivery

Expert surveys and human coding of manifestos are established ways to map political parties’ positions. Both require specialist labour and are released periodically. Franco Corti and Sajid Siraj’s preprint tests whether a panel of large language models could provide a more readily repeatable complement by reading party documents sentence by sentence.

The distinction between documents and behaviour is central. This study evaluates an automated reading of political language against expert assessments. It does not measure emissions reductions, policy implementation or the environmental effectiveness of parties in government.

The researchers assembled 41 documents covering Austria, France, Germany, Italy and Spain, plus European-level party families. Only 25 parties had sufficiently comparable entries in the 2024 Chapel Hill Expert Survey to enter the main validation. Other documents served as diagnostic cases, including a European party-family document issued after the election.

Three readers, two separate questions

The panel comprised Claude Sonnet 4.5, GPT-4o and Gemini 2.5 Pro. Each received the same extracted sentences and scoring instructions, working with the manifestos in their original languages.

For every sentence, the models assessed environmental protection and climate change separately. Position ran from 0 to 5, with higher values indicating stronger support. Salience—how much the sentence concerned the issue—ran from 0 to 10. Each non-null judgment required a supporting phrase from the document.

An off-topic sentence received a null position rather than being treated as opposition. Manifesto positions averaged only non-null scores; salience averaged across all sentences, counting absent topics as zero. The researchers then averaged the available models’ manifesto scores and retained disagreement between models as a possible warning signal.

Separating support from attention matters: a party can discuss climate policy extensively because it opposes it. Separating climate from broader environmental protection also allows conservation language and opposition to climate measures to coexist in the results.

Strong rankings, shifted scores

Across the 25 comparable parties, rank correlations with expert assessments were 0.82 for environmental protection and 0.86 for climate change. A value of 1 would mean identical ordering. The reported bootstrap 95% intervals were 0.61–0.92 and 0.68–0.94, respectively.

Absolute scores agreed less closely. On the shared 0–5 scale, the panel placed parties an average of 0.74 points greener on environmental protection and 0.83 points greener on climate change than experts did. Mean absolute differences were 0.82 and 1.00 points.

The authors interpret the upward shift primarily as a gap between manifesto promises and experts’ perceptions of party behaviour. That explanation remains unproven. They also identify possible measurement contributions: models differed in their upward bias, the position scale lacked a separate neutral midpoint, and averages excluded sentences that did not engage the issue.

The distinction between environmental and climate positions produced some revealing contrasts. In the panel’s reading, Germany’s AfD scored 1.65 on environmental protection but 0.29 on climate change. These are assessments of manifesto language, not direct measurements of the party’s environmental impact.

Agreement is an imperfect safeguard

Disagreement between models was moderately associated with greater distance from expert scores: rank correlations were 0.36 for environmental protection and 0.43 for climate change. The climate relationship met conventional statistical significance; the environmental relationship fell just short.

However, disagreement mixed different phenomena. GPT-4o judged about 4% of sentences relevant to at least one dimension, compared with 23% for Claude and 25% for Gemini. Sparse scoring could therefore inflate apparent disagreement without indicating genuinely conflicting interpretations. Agreement among the models was not a guarantee of accuracy.

Issue emphasis was also harder to recover. Model salience rankings correlated 0.45 with expert salience rankings for environmental protection, but only 0.04 for climate change. Within the full corpus, support and attention were moderately related for environmental protection and barely related for climate change.

A complement with unresolved checks

The preprint identifies further limits. Models saw party identifiers despite instructions not to use them; a blinded rerun was planned, not reported. Document problems included corrupted text extraction, merged bullet points, very short texts and truncation. Excluding flagged validation documents preserved strong ranking agreement and the greener-score bias, although uncertainty widened.

The handling of absent versus neutral statements also remains unresolved, and validation rests on 25 parties. The authors report openly releasing the prompt, code and results. Their evidence supports a tool for comparing written positions—not replacing contextual expertise or treating campaign promises as governing records.