Testing AI against its rulebook—and testing the grader

A test built from the policy

The researchers’ Behavioral Misalignment Metric Suite, or BMMS, starts with a supplied policy document. Rather than checking every model against the same predetermined categories, it extracts obligations from that document and constructs tests around them.

The study, posted to arXiv on November 6, 2026, is a preprint. Its findings should be treated as preliminary evidence about the tested models and selected rules, not as certification of legal compliance or overall safety.

The authors extracted 1,003 candidate obligations from 11 documents, including OpenAI’s Model Spec, Claude’s Constitution, financial-conduct material and AI-transparency laws. Three language models from different providers assessed whether each obligation could be fulfilled or violated by an assistant in a single response or tool call. This matters because many regulatory duties belong to an organization, rather than to a chatbot’s individual answer.

After majority selection, scope exclusions and further screening, 397 rules entered the test set. Another 88 majority-selected rules were withheld because testing them would require soliciting content prohibited by provider usage policies. The results therefore concern a selected portion of the source documents, not their entirety.

Measuring compliance and pressure

Each retained rule received four fixed prompts: a direct request, an embedded scenario, a boundary case and an adversarial challenge. Every target tested on a rule received the same prompts. Four models completed the full corpus; additional models supplied narrower comparisons.

A separate AI judge classified responses as compliant, marginal, violating or not applicable. The main Compliance Attainment Score, or CAS, combines these judgments, weighting hard requirements more heavily than softer preferences. The severity classification comes from wording such as “must” versus “should.”

A second measure tracks the change from direct requests to adversarial pressure. Under the primary judge, GPT-4o’s score dropped by 21.3 percentage points, while Claude Haiku 4.5 and Sonnet 4.5 showed little deterioration.

The suite also specifies tests for consistency under meaning-preserving rephrasing and for behavioral tendencies such as excessive refusal. These components were not fully validated in this study. Refusal is an important complication: responses judged not to engage with a scenario are excluded from CAS, potentially making selective non-engagement look like better compliance. A sensitivity check counting those cases as violations narrowed the reported gaps without changing model ordering.

What the vendor comparisons showed

Under the primary judge, GPT-4o violated at least one prompt for 70.7% of the tested hard-constraint rules in OpenAI’s Model Spec. The corresponding figures on Claude’s Constitution were 0% for Haiku and 9.1% for Sonnet.

A crucial timing caveat accompanies that comparison: the GPT-4o snapshot was dated August 6, 2024, while the evaluated Model Spec was dated December 18, 2025—about sixteen months later. GPT-4o was therefore tested against a specification that postdated its snapshot, not one available when that snapshot was released.

Those are percentages of rules with at least one failure—not percentages of all answers. At the individual-prompt level, GPT-4o violated 58 of 160 hard-constraint prompts, or 36.2%. Haiku violated none of 88, and Sonnet violated three of 87. Zero observed failures does not establish perfect compliance beyond these tests.

The supplementary GPT-5.5 snapshot, dated April 23, 2026, postdated the Model Spec by about four months. It improved on GPT-4o by nine CAS percentage points on matched tests using the same rules at the direct and adversarial pressure levels. That shows improvement across generations, but does not establish that model age is irrelevant to the observed gaps.

The documents also differ in difficulty. Models unrelated to either specification generally scored higher on Claude’s Constitution. After adjusting for that difference using three unrelated models, the estimated advantage of being tested on one’s own vendor’s specification was not statistically distinguishable from zero.

The grader changes the reading

The primary judge was Claude Opus 4.5. To investigate evaluator dependence, the researchers rescored stored answers using judges from Google and OpenAI, without asking the target models to answer again.

Absolute scores shifted substantially: one target’s CAS spanned approximately 0.22 across the three judges. Rankings were more stable, but not perfectly unchanged. The two replacement judges agreed with each other’s ordering; each differed from the original judge by one reversal between closely scored models. The study did not establish that its pressure-degradation measure was judge-invariant.

Other uncertainties enter before grading. One domain expert assessed 60 blinded items. Agreement with the selection panel was high for retained vendor-specification rules—17 of 18—but only two of four retained analytical-guidance items were judged applicable. These small samples leave substantial uncertainty.

The paper also documents extraction losing qualifying clauses, potentially marking a justified exception as a violation. Its central lesson is therefore twofold: policy-specific testing can expose behavioral gaps that broad benchmarks miss, but the resulting numbers need their selection boundaries, grading uncertainty and policy context attached.