AI forecasts of freelance-job success improve when models know the agent’s limits

A model’s confidence about an AI agent’s ability to complete a freelance job can change meaningfully when the agent’s constraints are spelled out. That is the central finding of PAYWORTHY, a preprint by Mohammad Reza Rezaei and colleagues at Upwork. Their experiment asks whether language models can identify promising assignments before an agent attempts them, rather than assess work after it is finished.

The researchers report a large difference between forecasts about a vaguely described, capable agent and forecasts informed by the actual working conditions. But their definition of success requires care: it is principally an AI judge’s verdict, not evidence that a customer accepted or paid for the output.

What the benchmark measures

PAYWORTHY contains 1,494 freelance postings across 12 broad categories, drawn through a multistage sampling procedure from 18.8 million postings spanning 2020 to mid-2026. An automatic filter retained jobs judged to have a defined deliverable rather than an ongoing service. The sampling procedure overrepresents some less common categories, so the benchmark is not a simple miniature of the marketplace.

The main agent, based on claude-opus-5, received the posting alone. It could not use tools, ask clarifying questions or obtain the client’s files and account access. It produced text or code from which deliverable files could be assembled. Although designed to work in a single response, long outputs were continued across multiple calls.

A gpt-5 judge read each posting, its budget and the complete deliverable. Its majority verdict across three votes determined whether the result was “pay-worthy”: work a client would probably pay the posted budget for without renegotiation. The judge accepted 54.4% of attempts.

Knowing the limits changes the ranking

Before any attempt, 18 models estimated the agent’s probability of success under two descriptions. The generic description presented an autonomous agent with a browser, code execution and file handling. The limits description instead specified six constraints, including working from the posting alone, having no tools and being unable to ask the client questions.

The researchers evaluated these forecasts using AUC, a ranking measure. An AUC of 0.5 corresponds to random ordering; higher values indicate better separation of accepted and rejected jobs. It is not the percentage of jobs predicted correctly.

With the generic description, AUC ranged from 0.45 to 0.56. With the limits description, it ranged from 0.66 to 0.86, and every forecaster improved. Among the highest-ranked fifth of jobs, limits forecasts selected work accepted 65–92% of the time, compared with 41–54% for generic forecasts.

Experiments with four forecasters suggested that two facts carried most of the improvement: the agent worked from the posting alone, and it operated in one response without tools. Neutral statements of similar length did not reproduce the gain.

Ranking and probability accuracy were nevertheless different problems. Limits forecasts were generally pessimistic: their average probabilities ranged from 7% to 34%, below the judge’s roughly 54% acceptance rate. Better ordering did not make those numbers ready-made thresholds for assigning work.

Missing files matter—but are not the whole explanation

Generic forecasts did not reliably favor self-contained jobs over assignments requiring client files. For an agent that never receives those files, that distinction is consequential.

However, the researchers also tested a more capable agent on 500 jobs, giving it web search, code execution and a simulated client that could supply information. The judge accepted 67% of its work, versus 52% for the posting-only agent on those same jobs. Generic forecasts still ranked outcomes near randomly, with a mean AUC of 0.47. This suggests that the mismatch between the generic description and the main agent does not fully explain the forecasting failure.

Approval is not payment

The limits advantage persisted under judges from four additional model families, although their acceptance rates differed substantially. Human evaluation also exposed a gap: two experienced raters assessed 149 items previously marked client-plausible or pay-worthy by the headline judge and accepted only about 35%. Limits forecasts ranked their verdicts better on average than generic forecasts.

That audit was limited: each category went to one rater, so inter-rater agreement could not be measured, and judge-rejected jobs were not evaluated.

The practical implication is narrower than a claim that models can predict freelance earnings. In this benchmark, explicit operational constraints improved advance ranking. Whether real clients would accept the agent’s work—and whether its prices would be fair—remains unanswered.