Plain text: 25-sec22-approval-models.md · on GitHub

§22 Machine learning can develop predictive models of human approval

Reframing Superintelligence: Comprehensive AI Services as General Intelligence — K. Eric Drexler, FHI Technical Report #2019-1 (Markdown edition). Section 22 of 40. Source label: approval-models. Rendered PDF: https://ora.ox.ac.uk/objects/uuid:9c05427a-6390-4b42-9c55-ee45f73a26ad

Thesis. By exploiting existing corpora that reflect human responses to actions and events, advanced ML systems could develop predictive models of human approval with potential applications to AI safety.

Cited by sections: §II, §3, §6, §8, §11, §14, §21, §23, §25, §26, §28, §29

Summary

Advanced ML capabilities will precede the development of advanced AI agents, and development of predictive models of human approval need not incur agent-related risks. Potential training resources for models of human approval include corpora of text and video that reflect millions of person-years of real and imagined actions, events, and human responses; potential corpora include news, history, fiction, law, philosophy, and more. Limited yet broad predictive models of human (dis)approval could provide commonsense defaults and constraints on both long-term plans and immediate actions. The challenges and potential applications of developing useful models of human approval suggest a range of topics for further consideration and inquiry.

Advanced ML technologies will precede advanced AI agents

The R&D-automation model of AI development shows how asymptotically-recursive AI-technology improvement could yield superintelligent systems (e.g., machine learning systems) without entailing the use of agents. In this model, agents are potential products, not necessary components.

Advanced ML can implement broad predictive models of human approval

As Stuart Russell has remarked, AI systems will be able to learn patterns of human approval “Not just by watching, but also by reading. Almost everything ever written down is about people doing things, and other people having opinions about it.”

By exploiting evidence from large corpora (and not only text), superintelligent-level machine learning could produce broad, predictive models of human approval and disapproval of actions and events (note that predicting human approval conditioned on events is distinct from predicting the events themselves). Such models could help guide and constrain choices made by advanced AI agents, being directly applicable to assessing intended consequences of actions.

Text, video, and crowd-sourced challenges can provide training data

Models of human approval can draw on diverse and extensive resources. Existing corpora of text and video reflect millions of person-years of actions, events, and human responses; news, tweets, history, fiction, science fiction, advice columns, sitcoms, social media, movies, CCTV cameras, legal codes, court records, and works of moral philosophy (and more) offer potential sources of training data. An interactive crowd-sourcing system could challenge participants to “fool the AI” with difficult cases, eliciting erroneous predictions of approval to enable training on imaginative hypotheticals.

Predictive models of human approval can improve AI safety

Predictive models of human approval and disapproval could serve as safety-relevant components of structured AI systems. Armstrong’s (2017) concept of low-impact AI systems points to the potentially robust value of minimizing significant unintended consequences of actions, and models of human approval imply models of what human beings regard as significant. When applied to Christiano’s (2014) concept of approval-directed agents, general models of human approval could provide strong priors for interpreting and generalizing indications of human approval for specific actions in novel domains. [fn: Note that learning predictive models of a human observer’s approval can enable an RL system to learn difficult tasks: see Christiano et al. (2017).]

Prospects for approval modeling suggest topics for further inquiry

The concept of modeling human approval by exploiting large corpora embraces a wide range of potential implementation approaches and applications. The necessary scope and quality of judgment will vary from task to task, as will the difficulty of developing and applying adequate models. In considering paths forward, we should consider a spectrum of prospective technologies that extends from current ML capabilities and training methods to models in which we freely assume superhuman capabilities for comprehension and inference.

In considering high-level capabilities and applications, questions may arise with ties to literatures in psychology, sociology, politics, philosophy, and economics. Models of approval intended for broad application must take account of the diversity of human preferences, and of societal patterns of approval and disapproval of others’ preferences.

Criteria for approval may be relatively straightforward for self-driving cars, yet intractable for tasks that might have broad effects on human affairs. For tasks of broad scope, the classic problems of AI value alignment arise, yet some of these problems (e.g., perverse instantiation) could be substantially mitigated by concrete models of what human beings do and do not regard as acceptable.