• “I think it would be interesting to try to develop a list of influential/common memes about AI, which are prevalent in different communities. (Examples: “Data is the new oil,” in certain policy communities, and “paperclippers,” in the EA x-risk community.) Then I think it’d also be interesting to ask whether any of these memes might be especially misleading or detrimental. This project could help people to better understand the worldviews of different communities better – and, I think, more importantly, to help people understand what kinds of communication/meme-pushing around AI governance might be most useful.” (Ben Garfinkel  – Some AI Governance Research Ideas)

80,000 Hours writes that ‘there are few AI policy practitioners with a technical AI background, leaving this perspective neglected.’

Mapping the technical possibilities of AI and assessing AI progress could be valuable – for more discussion and more specific questions see Alan Dafoe’s research agenda.

Other questions could come from this list from 80,000 Hours and these agendas from AI Impacts.

• “If we assume that AI software is similar to other software, what can we infer from observing contemporary software development? [concrete] For instance, is progress in software performance generally smooth or jumpy? What is the distribution? What are typical degrees of concentration among developers? What are typical modes of competition? How far ahead does the leading team tend to be to their competitors? How often does the lead change? How much does a lead in a subsystem produce a lead overall? How much do non-software factors influence who has the lead? How likely is a large player like Google—with its pre-existing infrastructure—to be the frontrunner in a random new area that they decide to compete in?” (AI Impacts)
• “How likely is it that AI systems will make it possible to cheaply, efficiently and reliably find vulnerabilities in computer systems with more skill than humans? What kinds of indicators might provide updates on this front? What measures could state or non-state actors take to prevent this coming about, and/or mitigate potential negative effects?” (80,000 Hours)
• “Compute is a very promising node for AI governance. Why? Powerful AI systems in the near term are likely to need massive amounts of compute, especially if the scaling hypothesis proves correct. Furthermore, compute seems more easily governable than other inputs to AI systems…Should governments set up such funds? Seeing as they are likely to be set up, how should they be designed?” (Markus Anderljung  – Some AI Governance Research Ideas)

Case studies on the development and governance of other transformative technologies can act as partial analogies for the development of AGI.

Possible questions include:

• “One set of possibilities for avoiding an AI arms race is the use of third party standards, verification, enforcement, and control…What are the prospects that great powers would give up sufficient power to a global inspection agency or governing body? What possible scenarios, agreements, tools, or actions could make that more plausible? What do we know about how to build government that is robust against sliding into totalitarianism and other malignant forms? What can we learn from similar historical episodes, such as the failure of the Acheson-Lilienthal Report and Baruch Plan, the success of arms control efforts that led towards the 1972 Anti-Ballistic Missile (ABM) Treaty,  and episodes of attempted state formation?” (Allan Dafoe’s research agenda)
• “This project explores the impact of US nuclear strategists on nuclear strategy in the early Cold War. What types of experts provided advice on US nuclear strategy? How and in what ways did they affect state policy making on nuclear weapons from 1945 through to the end of the 1950s (and possibly beyond)? How could they have had a larger impact?” (Waqar Zaidi – Some AI Governance Research Ideas)
• “History of existential risk concerns around nanotechnology: How did the community of people worried about nanotech go about communicating this risk, trying to address it, and so on? Are there any obvious mistakes that the AI risk community ought to learn from?; How common was it for people in the futurist community to believe extinction from nanotech was a major near-term risk? If it was common, what led them to believe this? Was the belief reasonable given the available evidence? If not, is it possible that the modern futurist community has made some similar mistakes when thinking about AI?” (Ben Garfinkel – Some AI Governance Research Ideas)
  
  

Examples of research exploring historical events to inform AI governance:

• Jade Leung, Who will govern artificial intelligence? Learning from the history of strategic politics in emerging technologies
• Cihon, Maas, and Kemp, Should Artificial Intelligence Governance be Centralised?: Design Lessons from History
• Zaidi and Dafoe, International Control of Powerful Technology: Lessons from the Baruch Plan for Nuclear Weapons
• Katja Grace, The Asilomar Conference: A Case Study in Risk Mitigation
• Preliminary survey of prescient actions – AI Impacts
• Katja Grace, Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation
• O’Keefe, How Will National Security Considerations Affect Antitrust Decisions in AI? An Examination of Historical Precedents
• AI Impacts, Historic trends in technological progress

Possible questions include:

• “Will EU regulation diffuse globally via the so-called “Brussels effect” (Bradford, 2020), or will there be a global race to the bottom with regards to minimum safety standards (Askell et al., 2019; Smuha, 2019)?” (Legal Priorities research agenda)
• “How should the scope of AI safety regulations be defined (Schuett, 2019)? Do we need new regulatory instruments (Clark & Hadfield, 2018)? How can compliance be monitored and enforced? Is there a need for stronger forms of supervision (Bostrom, 2019; Garfinkel, 2018)? If so, would they violate civil rights and liberties?” (Legal Priorities research agenda)
• “Is there a need to legally restrict certain types of scientific knowledge to prevent malevolent actors from gaining control over potentially dangerous AI technologies (Bostrom, 2017; Ovadya & Whittlestone, 2019; Shevlane & Dafoe, 2020; Whittlestone & Ovadya, 2020)? If so, how could this be done most effectively? To what extent is restricting scientific knowledge consistent with the relevant provisions of constitutional law?” (Legal Priorities research agenda)
  
  

See the Legal Priorities research agenda for more context and further questions. A survey of research questions for robust and beneficial AI from the Future of Life Institute also contains many questions law students could explore.

It may be valuable to explore topics such as “How can we avoid a dangerous arms race to develop powerful AI systems? How can the benefits of advanced AI systems be widely distributed? How open should AI research be?”

There are a lot of questions you can draw from Allan Dafoe’s AI governance research agenda. You could explore the potential impact of exacerbated inequality and job displacement on trends such as liberalism, democracy, and globalisation; who should be leading on AI governance; how substantial an advantage China has – as compared with other advanced developed (mostly liberal democratic) countries – in its ability to channel its large economy, collect and share citizen data, and exclude competitors; and how a dangerous arms race to develop AI could be prevented or ended.

Other potential questions include:

• “Taking into account likely types and applications of autonomous weapons, what are the likely effects on global peace and security?” (80,000 Hours)
• “What plausible paths exist towards limiting or halting the development and/or deployment of autonomous weapons? Is limiting development desirable on the whole? Does it carry too much risk of pushing development underground or toward less socially-responsible parties?” (80,000 Hours)
• “How might AI alter power dynamics among relevant actors in the international arena? (great and rising powers, developed countries, developing countries, corporations, international organizations, militant groups, other non-state actors, decentralized networks and movements, individuals, and others).” (Center for a New American Security)

These are some of the questions suggested in the Effective Altruism Psychology Lab’s research agenda. They suggest you reach out if you’re interested in pursuing research on any of these topics.

• “How do AI researchers think about risks from advanced AGI? How concerned are AI researchers about risks from uncontrolled AGI, and why? How do AI researchers’ views differ from those of other populations? Are they more or less concerned?” (Effective Altruism Psychology Lab)
• “How can we raise awareness of AGI risk among AI researchers? How do we create group dynamics and systems that ensure safe development of AGI?” (Effective Altruism Psychology Lab)
  • Seth Baum’s ‘On the promotion of safe and socially beneficial artificial intelligence‘ is one relevant paper.
• “What do people think about digital sentience? Do they believe that digital beings can be sentient? If not, why? Do people morally value or discount (different types of) potential digital beings?” (Effective Altruism Psychology Lab)
• What cognitive biases could undermine efforts to align AGI? Do people underappreciate how training an AGI on one task makes it competent to do another task? Do people succumb to the curse of knowledge when trying to predict AI’s ability to infer how humans want it to behave? (Psychology for Effectively Improving the Future)
  • Eliezer Yudkowsky’s ‘Cognitive Biases Potentially Affecting Judgment of Global Risks’ is one relevant paper.

Possible questions include:

• “Given what we are still learning about ourselves and our values, is it possible to anticipate the direction that our values are moving in, or the direction they should move in?” (AI Governance: A Research Agenda – Allan Dafoe)
• “If something like digital people became possible – digital beings that ought to have their own interests considered – who should have the right to create them, and how should voting work? (The default of “anyone can create as many digital people as they want, without limit” doesn’t seem ideal, especially assuming such digital people would vote.)” (Appendices for “Important, actionable research questions for the most important century” – Holden Karnofsky)
• “How should we value various possible long-run outcomes [of the development of AGI] relative to each other?” (Appendices for “Important, actionable research questions for the most important century” – Holden Karnofsky)
  
  

For more ideas, see:

• The post ‘Problems in AI alignment that philosophers could potentially contribute to’ and the comment discussion.
• A survey of research questions for robust and beneficial AI – the Future of Life Institute

Possible questions include:

• “How do AI researchers think about risks from advanced AGI? How concerned are AI researchers about risks from uncontrolled AGI, and why? How do AI researchers’ views differ from those of other populations? Are they more or less concerned?” (Effective Altruism Psychology Lab)
• “How can spread norms in favour of careful, robust testing and other safety measures in machine learning? What can we learn from other engineering disciplines with strict standards, such as aerospace engineering?” (Technical AI Safety Research outside of AI)
• “How can we best increase communication and coordination within the AI safety community? What are the major constraints that safety faces on sharing information (in particular ones which other fields don’t face), and how can we overcome them?” (Technical AI Safety Research outside of AI)
• “What capabilities might AI systems one day have, and what would be some possible social consequences? For example, what would happen if an AI system could generate language and images designed to persuade particular sets of people of particular propositions?” (80,000 Hours)