Keeping AI workflow instructions separate from implementation know-how

The knowledge missing from a checklist

A standard operating procedure, or SOP, describes how a task should be carried out. But instructions written for people often leave technical details unstated: where to retrieve a manager’s contact information, which software interface to call, or what to do when a service fails.

Mitsuhiro Nakayama and colleagues at IBM Japan investigate how to preserve those details when large language models generate agent skills—bundles of instructions, tools and potentially code—from SOPs. Their November 2026 arXiv preprint calls the maintenance problem “SOP-skill drift”: developers correct an initial implementation, but those corrections can disappear when a revised procedure produces a new skill.

The proposed solution is a separation of responsibilities, not a new language-model architecture. Keep the business procedure readable, store technical context separately, and use both when regenerating the implementation.

What belongs where?

The system captures knowledge from developers’ edits or clarification conversations. It then classifies that knowledge before deciding where to preserve it.

Process clarifications, business rules and important exceptions belong in the SOP. Technical details, such as software interfaces, library choices and timeout settings, belong in a separate contextual-knowledge document. Generated code and configuration form a third layer.

For example, whether an expense threshold includes the boundary amount is a business-rule question. How software retrieves the approver’s details is an implementation question. Separating them should allow a policy change without discarding still-useful technical knowledge.

Human involvement remains central. Users review proposed SOP updates; the framework does not establish that a model can safely decide organizational policy on its own.

Regeneration is not the same as execution

In the initial experiment described in the paper, researchers used three workflows: a synthetic expense-approval procedure and hiring and copy-editing procedures adapted from Indeed examples. A developer refined each initial generated skill to create a reference implementation. The system extracted knowledge from those changes, separated it into documents, then regenerated each skill three times without further human input.

Eight of nine regenerated skills were judged functionally equivalent to the references. The remaining case was considered partially equivalent, with a difference in implementation depth: the regenerated version included complete Python code while the reference remained a specification.

These are comparisons of generated artifacts and intended behavior, not a demonstrated rate of successful execution in a working business environment. The paper describes human judgment with model assistance for this initial study, while later material describes model-based judging for an expanded evaluation.

Why the judge changes the headline

The abstract reports an expanded study of eight Indeed-derived workflows using one language model. Across 24 trials, the reported functional-equivalence rate rises from 12% under a naive evaluation to 58% under a “contract-aware” evaluation intended to distinguish genuine requirement violations from differences resolved by contextual knowledge.

That 46-percentage-point increase should not be read as an equivalent improvement in operational task success. The evaluation itself changes.

A particularly revealing comparison removes a report labeling elements by their stability across generation runs. Without that report, the equivalence rate falls from 58% to 29%. The authors interpret this as evidence that contextual knowledge contributes value, but that the labels also amplify favorable judgments. Stability across model outputs is not the same as a human-validated requirement.

The contract-aware and naive judgments used the same model with different prompts. An attempted contract-aware cross-model check completed none of its 24 trials because of inference-service failures, leaving independent validation unresolved.

Promising reuse, unresolved maintenance

The abstract also reports that previously extracted knowledge covered 79.5% of newly introduced automation gaps under three procedure-revision perturbations. However, the supplied paper extract does not provide enough supporting detail to assess how those gaps or coverage were determined.

The initial study’s separated SOPs were 75–84% smaller than the combined SOP-plus-context documentation—not smaller than the original procedures. The refined SOPs actually grew, and generated skills could also become substantially longer. Separation relocates detail; it does not necessarily reduce total documentation.

The evidence supports investigating explicit knowledge reuse, not declaring the maintenance problem solved. The study is small, lacks longitudinal user studies, and does not address how to detect context that becomes obsolete or contradicts revised policy. As a preprint, it presents preliminary findings rather than established evidence of reliable workplace deployment.