Define the outcome. Know what counts as evidence.

The six tests

Each one has a factual answer. That is the whole design.

No test here asks how a platform feels, how modern it is, or how many integrations it has. Each asks something a competent engineer on the vendor's side can answer with a yes, a no, or a screen.

1. Does the AI hold standing privilege, or is privilege leased per operation and revoked on a clock?

   An AI running under a permanent administrator account is a permanent administrator, and everything else in the demo is downstream of that one fact. Ask them to show you the identity the action ran as, then ask whether that identity can do the same thing on a different machine right now. A pass looks like a grant scoped to one operation and one target, with an expiry that lands on the clock rather than on the work finishing. A dodge sounds like "it runs under our secure agent" — which names a product, not an identity.

   FOLLOW-UP · “Show me the account, and show me its expiry.” — see Just-in-Time Elevation

2. Can the model retrieve a credential value at any point in the flow?

   Redaction happens after the value has already been in the model's context. Masking happens after that. The only answer that survives contact with an auditor is that the value never reaches the model at all. Ask what the model receives when a step needs a password: a reference it cannot resolve, or the thing itself. A pass looks like a reference token crossing the boundary and a value that stops at it, with the connection authenticated before any generated script exists. A dodge sounds like "encrypted at rest and redacted from the logs" — a true statement about storage, offered in place of an answer about delivery.

   FOLLOW-UP · “Could I recover the plaintext from your prompt history?” — see Credential Vault

3. Who asserts the outcome — the tool, or the machine that was changed?

   "Job completed successfully" is the tool grading its own homework. The question is whether anything independent of the tool was asked, after the change, whether the thing is actually fixed. Ask them to break the fix again while you watch and re-open the same record. A pass looks like a fresh check executed on the target, whose answer is the field in the record — so a failure writes itself in without anyone deciding to report it. A dodge sounds like a status chip going green, or a summary paragraph written by the model about its own work.

   FOLLOW-UP · “Which line here came from the target, and which came from you?” — see Outcome Receipts

4. Was the rollback declared before execution, or reconstructed afterwards?

   Every platform can describe an undo once you ask for one. The test is whether it existed before the change did. Ask to see the reversal on screen before the run starts, and ask who wrote it — the person who certified the procedure, or the model, at the moment you asked. A pass looks like a reversal that is part of the procedure and a precondition of it being allowed to run at all. A dodge sounds like "you can always restore from backup", which is a different product answering a different question.

   FOLLOW-UP · “Run it, then undo it, on this call.” — see Automation & Runbooks

5. Is the human's authorization in the same record as the work?

   Approval in a ticketing system and execution in an automation platform, correlated by timestamp, is not a chain of custody — it is two stories that agree. Ask for one reference number that resolves to both the person who said yes and the job that ran. A pass looks like a single operation id carrying the named human, the time, the privilege, the job, and the result. A dodge sounds like "it is all in the audit log", at which point the only useful reply is: which one.

   FOLLOW-UP · “Give me one number I can quote back to you.” — see Commanded Autonomy

6. Can that record be edited afterwards without anything noticing?

   Evidence that an administrator can quietly correct is documentation, not evidence. Ask who can edit a record, who can delete one, and what visibly breaks when they do. A pass looks like records that reference each other, so changing one leaves every later record disagreeing with it, and a recipient can check that themselves without the vendor's help. A dodge sounds like "only administrators can modify the audit log" — which confirms the record can be modified.

   FOLLOW-UP · “If you edited one, how would I find out?” — see Security & architecture

Where the answers live

Every test is answered by a stage that already has to happen.

This is why the spec is architectural rather than a feature checklist: none of these six answers can be bolted on later. Each one is produced by a step in the operation, or it is not produced at all.

Our own column

How we answer, so you can check us against it.

Publishing a test you pass is easy and worth very little. What follows is only useful because each row names the mechanism and the page that explains it, so you can go after the mechanism rather than the answer.

We answer yes to all six — and every one of them is written so you can check it on a machine you control rather than take the answer from a website. Two things are worth knowing as you weigh it. SOC 2 Type II is on our roadmap and not yet in hand; where we stand, and the evidence we supply in the meantime, is on the security page. And this spec covers the architecture of an operation — it says nothing about breadth of coverage, which is a separate and fair question to put to us.

Two different instruments

The spec tests the architecture. The five questions test the answer.

They are used at different moments and they catch different things. Bring both.