Plain text: 27-sec24-human-oversight.md · on GitHub

§24 Human oversight need not impede fast, recursive AI technology improvement

Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 24 of 40. Source label: human-oversight. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad

Thesis. Human guidance and safety-oriented monitoring can operate outside core technology development loops, and hence are compatible with fast, recursive AI technology improvement.

Cites sections: §10 R&D automation dissociates recursive improvement from AI agency; §35 Predictable aspects of future knowledge can inform AI safety strategies

Cited by sections: §II, §4, §8, §11, §17, §21, §26, §28

Summary

It has been suggested that competitive pressures would favor fully automated AI technology development, minimizing human involvement in favor of speed, and potentially sacrificing safety. Technology development, however, differs from application development. Improvement of core research-enabling AI technologies (e.g., algorithms, architectures, training methods, and development infrastructure) need not be directly linked to applications, hence need not have direct, potentially problematic effects on the world. In considering human involvement in R&D, we can distinguish between participation, guidance, and monitoring. Here, participation acts within (and potentially delays) a process, guidance sets objectives for a process, and monitoring enables, for example, safety-oriented interventions. Both guidance and monitoring can operate outside core technology-development loops, hence need not impose delays; participation is optional, and can be a contribution rather than an impediment. Increasing development speed by relaxing in-the-loop human participation need not sacrifice guidance or safety-oriented monitoring. Automation of world-oriented application development presents different challenges and risks.

Must pressure to accelerate AI technology development increase risk?

Technical and economic objectives will continue to drive incremental yet potentially thorough automation of AI R&D. In considering asymptotically recursive automation of AI R&D, it is natural to think of ongoing human involvement as a source of safety, but also of delays, and to ask whether competitive pressures to maximize speed by minimizing human involvement will incur risks. “AI R&D”, however, embraces a range of quite different tasks, and different modes of human involvement differ in their effects on speed and safety. Understanding the situation calls for a closer examination.

Basic technology research differs from world-oriented applications

It is important to distinguish basic AI technology development from AI application development. Recursive technology improvement entails the development of AI technologies that support AI technology development; in this process, basic AI technologies (e.g., algorithms, training methods, ML components, and AI R&D tools) serve as both products and components of recursive R&D loops, while world-oriented application development operates outside those loops, building on their products. Thus, recursive R&D loops need not be directly linked to world-oriented applications, hence need not have direct effects on the world. When we ask whether human involvement might slow recursive improvement, we are asking a question about basic AI technology research; when we ask about risks, we are typically concerned with AI applications.

We can distinguish between human participation, guidance, and monitoring

In considering human involvement in R&D, we can distinguish between participation, guidance, and monitoring. Here, participation implies human actions within an R&D loop, potentially causing delays; guidance means setting objectives and assessing results (e.g., by evaluating the performance of new ML components in applications development) in order to orient research; monitoring means observation (e.g., of information flows, resource allocation, and the capabilities of delivered components) in order to prompt safety-oriented interventions.

Guidance and monitoring can operate outside the central AI R&D loop

Both guidance and monitoring operate outside the R&D loop for basic technology improvement: Guidance directs R&D toward valuable outputs, not by direct involvement in R&D loops, but by providing feedback signals that (for example) help to train operational R&D components or help R&D-management components allocate resources toward productive activities. Monitoring seeks to forestall potential risks, not by direct involvement in R&D loops, but by enabling interventions that forestall misbehavior mediated by R&D outputs. The nature of potential misbehaviors and requisite monitoring and interventions is outside the scope of the present discussion; policies would presumably be informed by contemporaneous, cumulative experience and safety research.

Fast, asymptotically-recursive basic research need not sacrifice safety

Because neither guidance nor monitoring operates inside an R&D loop, human involvement need not entail delays. Thus, if fully recursive technology improvement becomes both feasible and desirable, maximizing the rate of progress in basic AI technologies need not sacrifice potentially safety-critical human roles.

World-oriented applications bring a different range of concerns

Application development and deployment will typically have direct effects on the human world, and many applications will call for iterative development with extensive human involvement. World-oriented application development operates outside the basic-technology R&D loop, placing it beyond the scope of the present discussion.