How alien is an AI’s alien? A benchmark tests departures from familiar biology

A six-legged creature may sound alien, yet still follow a familiar blueprint: a head, a torso and limbs for walking. AlienBench, introduced in a November 6, 2026 arXiv preprint by Jonas Oppenlaender and Johanna M. Silvennoinen, tests whether language models move beyond such patterns. Its target is deliberately narrow: structural novelty in written descriptions of imaginary organisms, not creativity as a whole.

The researchers adapt a cognitive-psychology task in which people imagine creatures from another planet. Earlier work described how these inventions often retain familiar animal features. AlienBench turns that idea into explicit scoring rules, replacing a broad judgment of originality with separate decisions about biological characteristics.

Counting changes in the blueprint

The benchmark examines ten features: symmetry, sensing, movement, body plan, covering, reproduction, metabolism, communication, habitat and cognition. Each receives one point for departing from a specified default, or zero for retaining it. An unmentioned feature also receives zero.

The distinction concerns mechanism, not decoration. Six legs still count as leg-based locomotion; rolling or jet propulsion counts as a departure. Likewise, unusual skin colors do not qualify as a fundamentally different covering. These are operational reference points, not a claim that every Earth organism follows the same blueprint.

Three language-model judges independently assess each description. Their summed feature scores are averaged to produce a Ward Departure Score between zero and ten. The judges were Claude Opus 4.6, Gemini 3.1 Pro and GPT-5.4, using deterministic temperature settings.

The researchers evaluated ten generating models, five proprietary and five open-weight. Each produced 30 descriptions under each of six prompts, making 180 descriptions per model and 1,800 overall. Generation used temperature 1.0 with different random seeds. Each model’s overall score gives equal weight to the six conditions.

Those conditions included a simple alien-creature prompt, an explicit request to differ from Earth life, and a request for fuller biological detail. Three others specified darkness, ten times Earth’s gravity or liquid-ammonia oceans.

Prompting changes the outcome

The reported overall scores ranged from 3.28 for GPT-6 Luna to 6.55 for Ministral 3 14B 2512. Their bootstrap 95% confidence intervals were 2.87–3.70 and 6.19–6.89, respectively. Open-weight models averaged 5.40, compared with 4.58 for proprietary models. That is a result for these selected models and this task, not evidence that one model category is generally more creative.

No model led under every condition. Explicitly asking for maximum difference from Earth organisms pushed every model’s mean above 8.6. High gravity produced much lower scores: every model averaged 4.18 or less. Across models, ammonia oceans and requests for fuller detail increased scores relative to baseline, while darkness showed no detectable difference after correction for multiple comparisons.

Even modest rewording mattered. Across pairs of baseline-prompt wordings, the median difference in model scores was 1.21 points. Model rankings were only partly preserved. Comparisons therefore require fixed wording, rather than treating similar-sounding prompts as interchangeable.

What the scores cannot establish

Length is a central limitation. Longer descriptions have more opportunities to mention features and earn departure points. Because omissions score zero, AlienBench combines structural novelty with elaboration. The authors acknowledge that it evaluates delivered text, not creative ability isolated from verbosity.

A human comparison illustrates the problem. One hundred crowdsourced participants answered the baseline prompt, averaging 1.93, below all model baseline means, which ranged from 2.16 to 6.88. But writing effort and incentives differ between people and models. Undetected AI assistance also remains possible despite instructions against it. The authors treat the sample as a descriptive reference, not proof of superior machine imagination.

The judges agreed strongly on total scores, with Krippendorff’s alpha of 0.913, although agreement for sensory features was weaker at 0.422. Agreement measures consistency, not independent confirmation that judgments are correct; shared training material could contribute to common biases.

Most importantly, departing from familiar animal anatomy does not establish that an idea is newly invented. A model could reproduce science-fiction conventions and still score highly. The benchmark also does not establish coherence, biological plausibility or writing quality. Its contribution is a transparent, feature-level comparison under fixed rules—not a universal creativity ranking. The authors report releasing the prompts, generations, judge outputs and scoring pipeline for scrutiny.