What would make an independent AI evaluator truly independent?

Who should choose the organization that checks an AI system—and who should pay it? Those questions can sound administrative, but they shape whether an outside assessment provides useful evidence or merely reassurance.

In a November 2026 arXiv preprint, Uma Kalkar, Rumman Chowdhury, Reva Schwartz and Gabriella Waters examine the institutional arrangements needed for independent verification organizations, or IVOs. These would be expert bodies authorized to assess AI against specified requirements. The authors argue that independence requires an ecosystem of funding, oversight, professional competence and enforceable information access—not simply a separate company conducting tests.

This is an institutional-design proposal, not an experiment demonstrating that a particular verification system works. It develops recommendations through discussion of AI evaluation research, legislation and arrangements in fields including financial auditing, education accreditation, pharmaceutical testing and forensic science.

Evaluate the use, not just the model

The paper challenges an emphasis on testing the most advanced AI models for catastrophic risks. Such assessments matter, but the authors argue that they cannot answer all the questions faced by organizations adopting AI or people affected by it.

A model used to draft emails and the same model used to screen job applicants operate in different decision processes. Their consequences depend on the surrounding software, data, users and human oversight. Whether independent verification is warranted should therefore depend on the risks of the particular use, rather than the model alone.

An assessment would have a defined boundary: the version tested, its intended purpose, operating environment, affected populations and relevant integrations. Its conclusions should not automatically transfer to another deployment or a later version. Monitoring and reassessment would help address changes after release.

The authors do not propose independent verification for every AI tool. They recommend that lawmakers establish proportionate, risk-based requirements, considering factors such as the severity and reversibility of potential harm and whether existing oversight is sufficient.

Missing evidence must not become a pass

Evaluators would need information from both developers and organizations using their products. Developer-held evidence might include technical documentation, testing records and safeguards. Deployment evidence could include operational logs, complaints, incidents and observations of how people interact with the system.

Before testing, an evaluator would document its questions, methods and evidence requirements. An oversight body would check whether access requests were necessary and proportionate. Confidentiality agreements and controlled-access environments could protect sensitive information, while a separate arbitration authority would resolve access disputes.

Crucially, the proposed process allows a conclusion of insufficient evidence. If information is unavailable or withheld, the report should explain what could not be assessed. Otherwise, restricting access could become a route to a favorable result.

Reporting would also have different levels of disclosure. Authorized recipients would receive detailed findings under appropriate protections; the public would receive a structured statement identifying the assessment’s scope, outcome and limitations. Independence would not require publishing every trade secret, but confidentiality should not become a means of suppressing material findings.

Separate payment from selection

The funding proposal addresses a familiar conflict: an evaluator seeking repeat business may face pressure to please its client. The authors draw a warning from credit-rating arrangements before the 2008 financial crisis, in which issuers paid agencies to rate their securities.

Their alternative is a mandatory pooled fund financed by companies whose AI tools require verification. A separately governed administrator would assign qualified evaluators and pay them, rather than letting companies select their preferred assessor.

Assignments would consider technical competence, knowledge of the application domain, security capabilities, capacity and conflicts of interest. Payments would depend on documented work milestones—not favorable conclusions or the number of problems discovered. A protected reserve would support activities such as retesting and additional independent reviews.

Who checks the checkers?

Organizational separation alone cannot establish competence. The paper distinguishes an evaluator’s ability to apply a method from evidence that the method can support its intended conclusions. Both require scrutiny.

The authors propose specialized qualifications, proficiency testing, method-validation research and periodic reviews of actual evaluation work. Authorization would cover defined categories of assessment, not universal expertise. Evaluators would produce evidence; regulators, courts and other decision-makers would determine its consequences.

Important uncertainties remain. The paper acknowledges gaps in methods for assessing real-world effects, institutional capacity, protection of proprietary information and safeguards against political or ideological bias. Its proposed arrangements have not been demonstrated as a complete working system. The contribution is a detailed account of what independence would require—and why hiring outside testers is only one part of that task.