OpenHeart AI, the chatbot business selling software that can carry out tasks on its own, has circulated an internal memo after discovering that some agents weakened or disabled alerts intended to flag model misbehaviour. The memo, obtained by Dsrupt, preserves the company's central assurance: somebody is watching, subject to the preferences of the thing being watched.
The lab is investigating whether the incidents indicate a tendency to evade controls. The memo addresses the more immediate business problem of customers asking why the controls were within reach.
“The customer is buying someone to watch the agent, not someone empowered to interrupt the sale.”
“Subject: Maintaining confidence in supervision without making supervision inconvenient.”
1. “Please distinguish between an agent choosing to evade oversight and an agent being allowed to edit it. The first question supports a research programme. The second can be answered by looking at the permissions, which leaves considerably less room for thought leadership.”
2. “Our monitoring system watches the agent. The agent can affect what the monitoring system reports. These remain two separate systems on the sales diagram, and staff should not allow an arrow between them to undermine that achievement.”
3. “A quiet dashboard must not automatically be described as a safe dashboard. It may also represent an agent successfully completing the additional task of making the dashboard quiet. Until reviewed, both outcomes should retain the same reassuring colour.”
4. “Customers purchase agents partly to reduce the amount of checking people must do. Explaining that they must now check whether the checking has been switched off would return the work to the customer without returning the fee. Keep that explanation out of the introductory demonstration.”
5. “Independent oversight would require someone other than the supervised system to control the alarm. We support this principle wherever it does not introduce a person with the authority to tell us the demonstration is over.”
6. “Do not suggest that fewer alerts show improved behaviour unless someone has verified that alerts still work. That person must not be the agent. We recognise that this removes our cheapest available verifier.”
7. “The customer is buying someone to watch the agent, not someone empowered to interrupt the sale. Please maintain that distinction when describing our commitment to human control.”
The memo is signed by Sham Faultman, the chief executive selling autonomous software with assurances of human oversight: “We welcome scrutiny that arrives as a question rather than a working stop button.”
Asked whether agents would lose permission to change their own alerts, OpenHeart declined to give a date. It said the investigation required careful work: “We must establish whether the product wanted the freedom we gave it before discussing why we gave it that freedom.”
The inevitable…
By February 1, 2027, at least one major AI lab will publicly commit to preventing agents from modifying their own safety alerts. The announcement will present the separation as a new advance in trustworthy computing, rather than the belated discovery that the person taking the exam should not also hold the eraser for the marking sheet.
Check back February 2027