Why bigger AI models stumble on a specialized enterprise language

A conversational agent’s definition is more than a block of code: it specifies topics, permitted actions and behavioral guardrails. Salesforce’s AgentScript is a declarative language for expressing those choices. Because specialized enterprise languages are uncommon in public training data, success on familiar coding benchmarks may not predict how well a model can use them.

The authors introduce AgentScript-Eval, a benchmark of 100 tasks drawn from real agent-development work. The tasks ask a system to create an agent from a written specification, repair a broken definition or improve a working one. Programs are graded on whether they compile and whether the resulting agent meets executable checks in scripted, multi-turn dialogues. The checks assess behavior such as actions taken, conversation state and response constraints.

The study evaluates five models in direct-generation settings and examines agentic setups that can edit, compile and run programs against the checks. It reports strict success and partial credit: a task passes strictly only if all its checks succeed, while the pass rate averages the fraction of checks met. For direct generation, pass@3 counts a task as passed if any of three executions of the same generated program clears every check. For agentic runs, the paper reports pass@1, selecting the best outcome among programs evaluated during the agent’s trajectory. The model can receive feedback and revise its program along the way. These procedures give the two settings different opportunities to respond to execution results, so their strict scores are not a like-for-like comparison.

Direct generation proved difficult. The best model achieved 25% pass@3, with a 38% compilation rate. No model solved any of the 30 authoring tasks in the zero-shot setting. Tasks that supplied an existing program were more tractable: for the strongest model, zero-shot optimization reached 53.3%, while debugging reached 22.5%.

This contrast suggests that creating a complete definition from scratch was especially challenging in this benchmark. General programming ability may not provide reliable knowledge of a platform-specific language’s syntax and behavior. And compilation does not guarantee that an agent will behave as specified: some programs compiled but failed behavioral checks.

The results also show how performance varied with the surrounding setup. With Claude Opus 4.8, the agentic baseline recorded 33% strict success and a 46.5% partial-credit pass rate; direct generation with the same model recorded 25% pass@3 and a 31.3% pass rate. The agentic system could compile, execute and revise programs, and the reported outcome reflects the best program encountered during that process. Because direct generation instead runs one fixed program three times, the score gap cannot be attributed solely to iteration or read as a clean, directly comparable improvement.

Domain guidance produced another notable result. A static-documentation version of the skill pack scored 28% strict success; a live-plugin version scored 14%. These versions differed in both delivery and catalog breadth, so this comparison does not isolate the cause of the gap. The researchers then kept the plugin shell, 40-skill catalog, tool policy and bundled language server fixed while rewriting the authoring skill and restructuring its references for progressive disclosure. That configuration reached 37% strict success, a 61.1% partial-credit pass rate and a 98% compilation rate—the strongest reported configuration. Even there, behavioral success was incomplete: 40 of the 100 tasks received less than full credit. Better compilation did not ensure fully correct behavior.

Model size and price were not reliable guides to results in this test. The open-weight Gemma-4 31B model had the highest direct compilation rate, at 48%, and a 28.1% pass rate, near the strongest direct result of 31.3%. Its strict pass@3 score, however, was 15%, below the leading model’s 25%. These results show uneven rankings among the models tested, not that smaller models are generally as capable as larger ones.

The findings have a narrow scope: 100 tasks, one enterprise language and particular model-and-tool combinations. The paper notes that not every comparison isolates one factor, and the different scoring procedures limit comparisons between direct and agentic runs. The source describes the work as a NeurIPS 2026 workshop paper, but the supplied evidence does not establish that it was presented there. It is also available as an arXiv preprint; preprint status does not by itself establish peer review or scientific consensus. The study is best read as a focused evaluation of one enterprise coding problem, not a broad verdict on AI agents across business software.