# §24 Human oversight need not impede fast, recursive AI technology improvement

*Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 24 of 40. Source label: `human-oversight`. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad*

**Thesis.** Human guidance and safety-oriented monitoring can operate outside core technology development loops, and hence are compatible with fast, recursive AI technology improvement.

**Cites sections:** §10 *R&D automation dissociates recursive improvement from AI agency*; §35 *Predictable aspects of future knowledge can inform AI safety strategies*

**Cited by sections:** §II, §4, §8, §11, §17, §21, §26, §28

---
## Summary

It has been suggested that competitive pressures would favor fully automated AI technology development, minimizing human involvement in favor of speed, and potentially sacrificing safety. *Technology* development, however, differs from *application* development. Improvement of core research-enabling AI technologies (*e.g.*, algorithms, architectures, training methods, and development infrastructure) need not be directly linked to applications, hence need not have direct, potentially problematic effects on the world. In considering human involvement in R&D, we can distinguish between *participation, guidance,* and *monitoring.* Here, *participation* acts within (and potentially delays) a process, *guidance* sets objectives for a process, and *monitoring* enables, for example, safety-oriented interventions. Both *guidance* and *monitoring* can operate outside core technology-development loops, hence need not impose delays; *participation* is optional, and can be a contribution rather than an impediment. Increasing development speed by relaxing in-the-loop human participation need not sacrifice guidance or safety-oriented monitoring. Automation of *world-oriented application development* presents different challenges and risks.

## Must pressure to accelerate AI technology development increase risk?

Technical and economic objectives will continue to drive incremental yet potentially thorough automation of AI R&D. In considering [asymptotically recursive automation of AI R&D](→§10), it is natural to think of ongoing human involvement as a source of safety, but also of delays, and to ask whether competitive pressures to maximize speed by minimizing human involvement will incur risks. “AI R&D”, however, embraces a range of quite different tasks, and different modes of human involvement differ in their effects on speed and safety. Understanding the situation calls for a closer examination.

## Basic technology research differs from world-oriented applications

It is important to distinguish *basic AI technology* development from *AI application* development. Recursive technology improvement entails the development of AI technologies that support AI technology development; in this process, basic AI technologies (*e.g.*, algorithms, training methods, ML components, and AI R&D tools) serve as both products and components of recursive R&D loops, while world-oriented application development operates outside those loops, building on their products. Thus, recursive R&D loops need not be directly linked to world-oriented applications, hence need not have direct effects on the world. When we ask whether human involvement might slow recursive improvement, we are asking a question about basic AI technology research; when we ask about risks, we are typically concerned with AI applications.

## We can distinguish between human *participation, guidance,* and *monitoring*

In considering human involvement in R&D, we can distinguish between participation, guidance, and monitoring. Here, *participation* implies human actions within an R&D loop, potentially causing delays; *guidance* means setting objectives and assessing results (*e.g.*, by evaluating the performance of new ML components in applications development) in order to orient research; *monitoring* means observation (*e.g.*, of information flows, resource allocation, and the capabilities of delivered components) in order to prompt safety-oriented interventions.

## Guidance and monitoring can operate outside the central AI R&D loop

Both guidance and monitoring operate outside the R&D loop for basic technology improvement: *Guidance* directs R&D toward valuable outputs, not by direct involvement in R&D loops, but by providing feedback signals that (for example) help to train operational R&D components or help R&D-management components allocate resources toward productive activities. *Monitoring* seeks to forestall potential risks, not by direct involvement in R&D loops, but by enabling interventions that forestall misbehavior mediated by R&D outputs. The nature of potential misbehaviors and requisite monitoring and interventions is outside the scope of the present discussion; policies would presumably be informed by [contemporaneous, cumulative experience and safety research](→§35).

## Fast, asymptotically-recursive basic research need not sacrifice safety

Because neither guidance nor monitoring operates inside an R&D loop, human involvement need not entail delays. Thus, if fully recursive technology improvement becomes both feasible and desirable, maximizing the rate of progress in basic AI technologies need not sacrifice potentially safety-critical human roles.

## World-oriented applications bring a different range of concerns

Application development and deployment will typically have direct effects on the human world, and many applications will call for iterative development with extensive human involvement. World-oriented application development operates outside the basic-technology R&D loop, placing it beyond the scope of the present discussion.

## Further Reading

- §20 *Collusion among superintelligent oracles can readily be avoided*

- §21 *Broad world knowledge can support safe task performance*

- §22 *Machine learning can develop predictive models of human approval*

- §23 *AI development systems can support effective human guidance*

- §28 *Automating biomedical R&D does not require defining human welfare*

- §35 *Predictable aspects of future knowledge can inform AI safety strategies*