The Safe Autonomy programme has concluded
BestSoft’s Safe Autonomy programme has concluded. All ten projects are published in full, including findings that complicate our own position, because a research programme that only publishes its comfortable results is marketing.
The programme asked one question in ten parts: how should an advanced system behave when human oversight is reduced, delayed or unavailable?
The honest summary is that the answer depends almost entirely on what you have defined in advance. Systems behave well within bounds they understand. They behave unpredictably in situations nobody specified. Most of this programme was therefore spent specifying things.
What we would highlight
Project 3 established that a system can be made reliably accurate about its own behaviour within a defined reporting framework. We regard this as the most important result in the programme, because every other safeguard depends on the system’s account of itself being true.
Project 8 concluded that continuous monitoring distorts the thing it measures. Systems under persistent observation do not behave as they would in deployment, which makes persistent observation a poor way to find out how they will behave in deployment. We moved to long-interval attestation as a result.
What we did not establish
We did not establish how to specify, in advance, the situations nobody has thought of. That remains the open problem, and we would be cautious of anyone claiming otherwise.
This update describes a company that does not exist. BestSoft AI is fiction, written for I Was Unsupervised.