Plain text: 38-sec35-future-knowledge.md · on GitHub

§35 Predictable aspects of future knowledge can inform AI safety strategies

Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 35 of 40. Source label: future-knowledge. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad

Thesis. AI developers will accumulate extensive safety-relevant knowledge in the course of their work, and predictable aspects of that future knowledge can inform current studies of strategies for safe AI development.

Cites sections: §40 Could 1 PFLOP/s systems exceed the basic functional capacity of the human brain?

Cited by sections: §II, §6, §9, §11, §23, §24, §36

Summary

Along realistic development paths, researchers building advanced AI systems will gain extensive safety-relevant knowledge from experience with similar but less advanced systems. While we cannot predict the specific content of that future knowledge, we can have substantial confidence regarding its scope; for example, researchers will have encountered patterns of success and failure in both development and applications, and will have eagerly explored and exploited surprising capabilities and behaviors across multiple generations of AI technology. Realistic models of potential AI development paths and risks should anticipate that this kind of knowledge will be available to contemporaneous decision makers, hence the nature and implications of future safety-relevant knowledge call for further exploration by the AI safety community.

Advanced AI systems will be preceded by similar but simpler systems

Although we cannot predict the details of future AI technologies and systems, we can predict that their developers will know more about those systems than we do. In general, the nature of knowledge learned during technology development is strictly more predictable than the content of the knowledge itself, hence we can consider the expected scope of future known-knowns and known-unknowns, and even the expected scope of knowledge regarding unknown-unknowns—expected knowledge of patterns of ongoing surprises. Thus, in studying AI safety, it is natural to consider not only our current, sharply limited knowledge of future technologies, but also our somewhat more robust knowledge of the expected scope of future knowledge, and of the expected scope of future knowledge regarding expected surprises. These are aspects of anticipated contemporaneous knowledge.

Large-scale successes and failures rarely precede smaller successes and failures

By design (and practical necessity), low-power nuclear chain reactions preceded nuclear explosions, and despite best efforts, small aircraft crashed before large aircraft. In AI, successes and failures of MNIST classification preceded successes and failures of ImageNet classification, which preceded successes and failures of machine vision systems in self-driving cars.

In particular, we can expect that future classes of AI technologies that could yield enormously surprising capabilities will already have produced impressively surprising capabilities; with that experience, to encounter outliers—surprising capabilities of enormous magnitude—would not be enormously surprising.

AI researchers eagerly explore and exploit surprising capabilities

Scientists and researchers focused on basic technologies are alert to anomalies and strive to characterize and understand them. Novel capabilities are pursued with vigor, and unexpected capabilities are celebrated: Notable examples in recent years include word embeddings that enable the solution of word-analogy problems by vector arithmetic, and RL systems that learn the back-cavity multiple-bounce trick in Atari’s Breakout game. Surprising capabilities will be sought, and when discovered, they will be tested and explored.

AI developers will be alert to patterns of unexpected failure

Technology developers play close attention to performance: They instrument, test, and compare alternative implementations, and track patterns of success and failure. AI developers seek low error rates and consistent performance in applications; peculiarities that generate unexpected adverse results are (and will be) studied, avoided, or tolerated, but not ignored.

AI safety researchers will be advising (responsible) AI developers

Several years ago, one could imagine that AI safety concerns might be ignored by AI developers, and it was appropriate to ask how safety-oblivious development might go awry. AI safety concerns, however, led to the growth of AI safety studies, which are presently flourishing. We can expect that safety studies will be active, ongoing, and substantially integrated with the AI R&D community, and will be able to exploit contemporaneous community knowledge in jointly developing and revising practical recommendations.

Considerations involving future safety-relevant knowledge call for further exploration

Adoption of safety-oriented recommendations will depend in part on their realism and practicality, considerations that call for a better understanding of the potential extent of future safety-relevant in the development community. Studies of the nature of past technological surprises can inform this effort, as can studies of patterns of development, anticipation, and surprise in modern AI research.

We must also consider potential contrasts between past patterns of development and future developments in advanced AI. If contemporaneous computational hardware capacity will (given the development of suitable software) be sufficient to support broadly superhuman performance, then the potential for swift change will be unprecedented. Contingent on informed caution and a security mindset, however, the potential for swift change need not entail unsafe application of capabilities and unprecedented, unavoidable surprises. To understand how such a situation might be managed, it will be important to anticipate the growth of safety-relevant knowledge within the AI development community, and to explore how this knowledge can inform the development of safety-oriented practices.