AI-generated survey answers can preserve group patterns while missing individuals

Large language models can generate survey answers from descriptions of people, but what would make those answers useful research data? A preprint by Enrico Angelo Raffaele D’Ecclesiis, Fulvio Castellacci and Arash Ahmadi argues that the answer depends on what researchers intend to do with them.

Their analysis finds no single winner across all measures. Conventional supervised machine learning predicted individuals’ responses more accurately. Some language models, however, better preserved selected features of the overall response distribution, while several reproduced demographic ordering and parts of the geographic and predictive structure. These are preliminary findings from a preprint, not a general validation of artificial survey participants.

Three different tests of usefulness

The proposed framework separates three questions. Predictive validity asks whether a model matches the observed answers of the particular people whose profiles it receives. Distributional validity asks whether generated answers reproduce the overall average, spread and shape of real responses. Substantive validity asks whether analysis of the synthetic answers preserves relationships such as differences between demographic groups.

These requirements are not interchangeable. Filling in an individual’s missing response demands different evidence from exploring group comparisons. The authors therefore avoid combining the dimensions into a universal score that could conceal important failures.

Recreating answers, not a population

The researchers used Special Eurobarometer 554, surveyed across the European Union’s 27 member states in April and May 2024. They restricted the analysis to people aged 18–65 who were not classified as retired, then required complete outcome and profile information.

The final samples contained 10,935 respondents for a broad AI-and-digital-technology attitude index, and 12,583 for each of two other indices: perceived effects associated with AI and attitudes toward specific workplace applications. Each index averaged questionnaire items on a scale from one to four, with higher scores indicating more favorable attitudes.

Claude Sonnet 4.6, GPT-4.1 mini, Gemini 3 Flash and Grok 4.1 Fast Non-Reasoning received profiles containing demographic, household, employment, economic and attitudinal information. The profiles excluded answers used to construct the target indices. The models generated item-level responses without task-specific fine-tuning or supervised calibration to the survey outcomes.

Four supervised methods—Elastic Net, Random Forest, Histogram Gradient Boosting and Super Learner—provided a reference using the same substantive profile information. Unlike the language models, these methods learned from observed survey answers. Their reported predictions used ten-fold out-of-fold evaluation, so each respondent was predicted by a fitted model that excluded that respondent.

Accuracy and realism diverged

Random Forest had the lowest individual prediction error for all three outcomes. For general attitudes, its root mean squared error was 0.496 points on the four-point scale, compared with 0.540 for Claude, the strongest language model on this measure. For concrete workplace uses, the corresponding errors were 0.550 and 0.661.

But supervised predictions had only roughly half the observed standard deviation. This compression is consistent with predicting conditional averages: a good estimate of someone’s expected answer need not reproduce all the variation in actual answers.

GPT preserved more of that spread for general attitudes, reaching about 85% of the observed standard deviation, and achieved the lowest Wasserstein distance, a measure of distributional mismatch. More variation was not automatically better, however. Grok’s general-attitude responses were more dispersed than the real answers but also biased downward.

Specific workplace applications proved particularly difficult. All four language models generated less favorable average attitudes than observed, with downward biases ranging from about 0.26 to 0.69 points.

Preserved patterns still need scrutiny

For the general-attitude and effects indices, correlations between observed and synthetic age-group and education-group means were mostly above 0.98. Yet the models often exaggerated the size of group differences. Preserving their ordering did not guarantee accurate levels or gaps.

Country-level agreement also varied. Gemini’s general-attitude country means correlated 0.727 with observed means. For workplace-use attitudes, Claude reached 0.653, whereas GPT reached only 0.185.

To compare predictor structure, the researchers fitted Elastic Net models to observed and fixed synthetic outcomes, removing predictor blocks in turn and measuring the deterioration in prediction. Some rankings were similar, but these were predictive associations—not evidence that causal effects were preserved.

A bounded result

The study covers one survey, one topic and particular models, profiles and prompts. It inherits real respondents’ characteristics, so it does not test whether a language model can independently generate a representative population. Alternative temperature settings produced limited qualitative changes for three models; Claude was not included in that check.

The practical conclusion is narrower than replacing human surveys: synthetic answers may preserve properties useful for a specific task, but those properties must be tested directly against human evidence. Success on one dimension cannot establish suitability for another.