What crossword constraints reveal about AI riddle answers

A riddle about a familiar drink might accept both “Coke” and “Coca-Cola.” An evaluator expecting only one can mark the other wrong. Cultural riddles make this a useful problem to study: their clues draw on knowledge tied to particular communities, and may admit different valid interpretations or names.

The CulturalCrosswords study asks how constraints change large language models’ answers—and whether those answers fit together in a crossword. It distinguishes matching a saved reference answer from giving a listed alternative, agreeing at shared grid cells and placing answers correctly. These measures overlap, but they do not test the same thing.

Same clues, different constraints

Researchers built 53 crosswords from crowdsourced riddles, covering 42 languages and 42 countries. They evaluated 11 model configurations on the same clues in five formats: a plain batch; clues with answer lengths; lengths with partial-answer hints; crossword information with answers returned as a list; and a completed grid.

This matched design helps compare formats, but they are not simply steps on one ladder. Hints directly reveal some correct answer units; crossings require answers to agree with one another. The grid task also changes the requested output. The researchers scored answers against the saved solution and, for some clues, separately credited a set of accepted alternatives. They checked crossing compatibility and whether answers appeared in the correct grid positions.

Puzzle sets were weighted equally in the main comparisons. Uncertainty intervals reflect variation across language groups in this collection, not how much a model might vary across repeated attempts.

Lengths help—but what does “help” mean?

For 962 clues with listed alternatives, adding lengths raised agreement with the reference answer by 22.8 percentage points. When the scoring also credited listed alternatives, the gain was 8.4 points. The difference suggests some of the apparent improvement came from selecting the required answer form, rather than necessarily resolving what the clue meant.

One example in the paper: after receiving a length, a model switched from “COCACOLA” to “COKE.” If “COKE” is the saved answer, reference-only scoring rewards the change, even though both names identify the same drink. Other changes may reflect interpretation rather than wording. The alternative list is incomplete, so neither scoring method establishes how often models truly resolved ambiguity.

Adding partial hints to lengths brought further gains: 9.6 points on reference-only scoring and 8.4 points when listed alternatives counted. Adding crossword information while still requesting an answer list raised those scores by 4.6 and 3.0 points, respectively. These results show improvements on the benchmark’s scoring measures, but do not reveal exactly how models reasoned or establish that every credited answer was culturally appropriate.

Fitting together is another skill

With crossword information, the share of all crossings where answers were compatible rose from 61.8% to 71.5% compared with lengths alone—a 9.7-point increase. Reference agreement rose by 2.8 points in the same comparison. Compatibility can therefore capture something an answer-key score alone misses: answers may fit together more often without a similar-sized increase in reference matches.

Compatibility is not proof of correctness. Two wrong answers can agree at a shared cell, and an incorrect answer can constrain another in the wrong direction. The study measures outputs, not internal reasoning, so it cannot show how much models used crossings to solve clues.

Returning an answer list and filling the grid were also distinct tasks. One model’s reference agreement fell from 75.6% with a crossword answer list to 5.4% when asked to fill the grid. Another scored similarly on the two formats after recovery attempts. These gaps cannot be attributed solely to placement: instructions and execution differed. The researchers also found that grid structure could be valid without the grid containing the most correct answers.

What the results cannot establish

This benchmark is a selected set, not a representative survey of languages or cultures. Some riddles could not be fitted into connected grids; some writing systems lacked the required segmentation procedure; and some clues were excluded after review. Language, cultural content and puzzle construction are therefore entangled. Results for a writing-system group or individual language should not be treated as a general measure of model performance in that language.

Model comparisons have limits, too. Three models received extra attempts with clearer output instructions for selected incomplete answer lists. This recovery procedure changed some results, and the study does not compare models at equal computational budgets. It analyzes one retained response per model, puzzle and condition, so its uncertainty intervals do not represent repeated-run variability.

The paper is an arXiv preprint; its presence there does not establish independent peer review. Although the authors describe reproducibility materials, they also note that reference answers and alternative-answer judgments are not independently validated across the full benchmark. The findings are best read as a diagnostic of these puzzles and evaluation choices—not a verdict on models’ cultural understanding. The narrower lesson is that evaluations should distinguish matching an answer key, giving an acceptable answer, fitting shared constraints and completing the requested format.