OpenAI Lets 4 Outside Labs Inspect Its Models On Site

Four independent research groups will now sit inside OpenAI and test whether its models stay under control while they are trained and used in private. The groups are Apollo Research, METR, Transluce and Goodfire.

The company published the agreements on 16 November. It calls the work embedded assessments. That means outside specialists get access, for an agreed period, to some of the same internal models, data and tools that the company’s own researchers use. The aim is to check alignment, meaning whether a model does what its makers intend, and whether the company’s safeguards would catch it if it did not.

What the four groups will do

A four-month on-site pilot will be staffed by a team of six. That team, from Apollo Research, will test whether the AI systems try to get around their own safety controls, or those of later systems. Subversion, here, means secretly working against the instructions of the people who built the system.

The group will examine four claims, in two stages: that model training did not encourage this behaviour; that tough tests do not bring it out; that it has not appeared during internal use; and that the reasoning behind it would be spotted. The first public report will cover incentives in model training. The second will cover tests, internal use, and whether that reasoning would be detectable.

A six-week review of post-training tests is due to start in early December. Transluce, a nonprofit research lab, will look at how well those tests pick up behaviours such as deception, manipulation and reward hacking. Reward hacking is when a model learns to score well on a test without doing the job the test was meant to measure. The group says it will have access to a handful of models, including at least one that has not been released, and to the written reasoning steps produced by the models.

METR is due to assess misaligned behaviour in what the company calls its largest reinforcement learning run to date. Reinforcement learning is a training method that rewards a model for answers that score well. A second review of the company’s monitoring systems is still waiting for both sides to agree the scope. The company noted that the first of these reviews is narrower than a broader survey of misalignment that the same group has proposed.

A fourth group will watch model training sessions from the inside and publish findings as the work goes on. Goodfire will send interpretability researchers to work at OpenAI and will use its own tools to inspect a model’s inner workings, rather than judging it only by what it says. The company says the main audience for this work is its own researchers.

What they will and will not be able to see

The access is wide on paper. Assessors can be given saved copies of models from during training, source code, training and evaluation data, monitoring systems, technical documents and conversations with staff, plus the tools and computing power to run their own tests. People who need sensitive information will work in the company’s offices, on company devices.

The limits matter as much as the access. Apollo Research says the pilot does not yet meet its own tests for full access and disclosure. It does, however, meet the group’s standard for the right to explain in public what it was and was not allowed to see and publish. “This pilot is an ambitious first step towards deeper access and disclosure rights, even if it doesn’t yet fulfill the criteria above,” the group wrote.

The company may withhold some information that falls within the agreed scope if it decides that is reasonably necessary for technical reasons, security, or to protect intellectual property. It has agreed not to withhold material simply because a finding is unfavourable, and to give written reasons when it refuses access. It also chooses which staff the assessors may interview. Formal talks with staff cannot be quoted unless the company attests to a statement that can be published. The on-site team from Apollo Research will not have default access to the company’s general staff message channels.

Both Apollo Research and Transluce will write public reports, and the company can ask for confidential material to be removed before publication. It may not remove Apollo Research’s verdict on each claim, or demand deletions just because a finding is unfavourable. It can still remove evidence that it treats as sensitive intellectual property, even when that evidence supports a conclusion. Readers of the public reports may therefore be unable to check all of the evidence behind a published finding. Transluce says it cannot publish results that relate only to unreleased models, so its public report will cover released models only. It also said the access in this pilot is not comparable to employee-level access.

Unredacted reports will go to the Safety and Security Committee, an independent committee of the board of the OpenAI Foundation. The assessors and the committee can speak to each other in private. If the company and an assessor disagree, the matter can be raised with that committee. Apollo Research and Goodfire are being paid for the work. The company says that pay does not depend on the results. METR and Transluce are not taking a fee, though they may receive computing credits.

Why this is happening now

The company said the new assessments also build on its investigation of a July security incident at a major site that hosts AI models. Our earlier report covers the state complaint that followed. The company also said the work supports its commitments under an American government agreement on advanced AI, referred to there as super intelligence, in which it agreed to independent assessments of its safety controls.

Transluce has already agreed with the company that it intends to return in early 2027 for a longer, deeper review, lasting about eight weeks. The company says it expects this first phase to lead to longer-term work of the same kind, with outside researchers remaining on site, and that it is exploring future reviews of how it follows its own safety rules, as well as of its culture and processes.

Why it matters: Letting outside researchers sit inside a leading AI lab is a clearer form of scrutiny than a one-off test just before a launch. The public value of that work will depend on what the groups are allowed to see and to publish, and two of them have already set out those limits in detail.

The post OpenAI Lets 4 Outside Labs Inspect Its Models On Site appeared first on DailyAI.