Interested in working on this research direction? Apply for our coaching
Start here for an introduction to existential risk research. Read more
This profile is tailored towards students studying communications, media and marketing, computer science, economics, history, law, political science, psychology and cognitive sciences, philosophy and ethics and sociology. We expect there to be valuable open research questions that could be pursued by students in other disciplines.
Why is this a pressing problem?
Artificial intelligence is becoming increasingly powerful. AI systems can solve college-level maths problems, beat champion human players at multiple games and generate high quality images. They can be used in many ways that could help humanity, for example by identifying cases of human trafficking, predicting earthquakes, helping with medical diagnosis and speeding up scientific discovery.
The AI systems described above are all ‘narrow;’ they are powerful in specific domains, but they can’t do most tasks that humans can. Nonetheless, narrow AI systems present serious risks as well as benefits. They can be designed to cause enormous harm – lethal autonomous weapons are one example – or they can be intentionally misused or have harmful unintended effects, for example due to algorithmic bias.
It seems likely that at some point, ‘transformative AI’ will be developed. This phrase refers to AI that ‘precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution.’ One way this could happen is if researchers develop ‘artificial general intelligence;’ AI that is at least as intelligent as humans across all domains. AGI could radically transform the world for the better and help tackle humanity’s most important problems. However, it could also do enormous harm, even threatening our survival, if it doesn’t act in alignment with human interests.
Work on making sure transformative AI is beneficial to humanity seems very pressing. Multiple predictions (see here, here and here) suggest that transformative AI is likely within the next few decades, if not sooner. A majority of experts surveyed in 2022 believed there was at least a 5% chance of AI leading to extinction or similarly bad outcomes, while a near majority (48%) believed there was at least a 10% chance. Working on preventing these outcomes also seems very neglected – 80,000 Hours estimates that 1,000 times more money is being spent on speeding up the development of transformative AI compared to the money spent on reducing its risks.
AI governance research is one way the development and use of AI could be guided towards more beneficial outcomes. This is research that aims to understand and develop ‘local and global norms, policies, laws, processes, politics and institutions (not just governments) that will affect social outcomes from the development and deployment of AI systems.’ It can include high level questions such as how soon AGI will be developed, how it will affect the geopolitical landscape, and what ideal AI governance would look like. It can also include researching the possible impacts of AI on specific areas such as employment, wealth equality and cybersecurity, and developing specific solutions – such as lab policies to incentivise responsible research practices.
Watch the conference talk below in which Alan Dafoe discusses the space of AI governance for more information.
Explore existing research
- This online curriculum on AI governance is a great place to start if you want to learn more about this area and find relevant papers.
Other useful reading lists:
- Reading Guide for the Global Politics of Artificial Intelligence – Allan Dafoe
- Governance of AI – some suggested readings – Ashwin Acharya
Research organisations focused on current and near-term impacts of AI include:
- The Future of Life Institute; The Brookings Institute; RAND Corporation; Data & Society; Amnesty International; the Alan Turing Institute; Stockholm International Peace Research Institute; OECD; the World Economic Forum; the Center for Long-term Cybersecurity; the Wilson Center; Center for Security and Emerging Technology; Partnership on AI; The Global Partnership on Artificial Intelligence; CEPS; Digital Europe; Bruegel
Research organisations doing research related to risks from AGI include:
- Center on Long-term Risk; Global Catastrophic Risk Institute; Centre for the Study of Existential Risk; AI Objectives Institute; The Centre for Long-term Resilience; Partnership on AI; Leverhulme Centre for the Future of Intelligence; The AI Objectives Institute; Median Group; Convergence Analysis; Centre for the Governance of AI; the Future of Humanity Institute; The Future Society; Rethink Priorities; Epoch; The Future of Life Institute; Open Philanthropy; AI Impacts; AI Security Initiative; Center for Security and Emerging Technology
- Important, actionable research questions for the most important century – Holden Karnofsky
- A survey of research questions for robust and beneficial AI – Future of Life Institute
- Promising research projects – AI Impacts
- Possible Empirical Investigations – AI Impacts
- Some AI Governance Research Ideas – Markus Anderljung & Alexis Carlier
- Guide to working in AI policy and strategy – Concrete questions in AI policy we need to answer – 80,000 hours
- Technical AI Safety Research outside of AI – Richard Ngo
- Psychology for Effectively Improving the Future – Effective Altruism Psychology Lab
- Artificial Intelligence and Global Security Initiative Research Agenda – Center for a New American Security
- Legal Priorities Research: A Research Agenda – The Legal Priorities Project
- AI Governance: A Research Agenda – Allan Dafoe
Find a thesis topic
If you’re interested in working on this research direction, below are some ideas on what would be valuable to explore further. If you want help refining your research ideas, apply for our coaching!
- “I think it would be interesting to try to develop a list of influential/common memes about AI, which are prevalent in different communities. (Examples: “Data is the new oil,” in certain policy communities, and “paperclippers,” in the EA x-risk community.) Then I think it’d also be interesting to ask whether any of these memes might be especially misleading or detrimental. This project could help people to better understand the worldviews of different communities better – and, I think, more importantly, to help people understand what kinds of communication/meme-pushing around AI governance might be most useful.” (Ben Garfinkel – Some AI Governance Research Ideas)
80,000 Hours writes that ‘there are few AI policy practitioners with a technical AI background, leaving this perspective neglected.’
Mapping the technical possibilities of AI and assessing AI progress could be valuable – for more discussion and more specific questions see Alan Dafoe’s research agenda.
Other questions could come from this list from 80,000 Hours and these agendas from AI Impacts.
- “If we assume that AI software is similar to other software, what can we infer from observing contemporary software development? [concrete] For instance, is progress in software performance generally smooth or jumpy? What is the distribution? What are typical degrees of concentration among developers? What are typical modes of competition? How far ahead does the leading team tend to be to their competitors? How often does the lead change? How much does a lead in a subsystem produce a lead overall? How much do non-software factors influence who has the lead? How likely is a large player like Google—with its pre-existing infrastructure—to be the frontrunner in a random new area that they decide to compete in?” (AI Impacts)
- “How likely is it that AI systems will make it possible to cheaply, efficiently and reliably find vulnerabilities in computer systems with more skill than humans? What kinds of indicators might provide updates on this front? What measures could state or non-state actors take to prevent this coming about, and/or mitigate potential negative effects?” (80,000 Hours)
- “Compute is a very promising node for AI governance. Why? Powerful AI systems in the near term are likely to need massive amounts of compute, especially if the scaling hypothesis proves correct. Furthermore, compute seems more easily governable than other inputs to AI systems…Should governments set up such funds? Seeing as they are likely to be set up, how should they be designed?” (Markus Anderljung – Some AI Governance Research Ideas)
There are a lot of questions you can draw from Alan Dafoe’s AI governance research agenda. For example, you could explore the impact of exacerbated inequality and job displacement on trends such as liberalism, democracy, and globalisation; to what extent countries are able to internalise the returns on their AI investments, or whether talent inevitably gravitates towards and benefits the existing leaders in AI (e.g. Silicon Valley).
Other potential questions include:
- “How plausible are safety concerns about economic dominance by influence-seeking agents, as well as structural loss of control scenarios? Can these be reformulated in terms of standard economic ideas, such as principal-agent problems and the effects of automation?” (Richard Ngo’s post Technical AI Safety Research outside of AI)
- “Which tasks will there be most economic pressure to automate, and how much money might realistically be involved?” (Richard Ngo’s post Technical AI Safety Research outside of AI)
- “What are the biggest social or legal barriers to automation?” (Richard Ngo’s post Technical AI Safety Research outside of AI)
- “Estimate the change in inputs in mathematicians, scientists, or engineers, as a complement to estimates for rates of progress in those fields.” (Possible Empirical Investigations – AI Impacts)
A set of open questions and introduction to the field can also be found in The Economics of Artificial Intelligence: An Agenda book.
Case studies on the development and governance of other transformative technologies can act as partial analogies for the development of AGI.
Possible questions include:
- “One set of possibilities for avoiding an AI arms race is the use of third party standards, verification, enforcement, and control…What are the prospects that great powers would give up sufficient power to a global inspection agency or governing body? What possible scenarios, agreements, tools, or actions could make that more plausible? What do we know about how to build government that is robust against sliding into totalitarianism and other malignant forms? What can we learn from similar historical episodes, such as the failure of the Acheson-Lilienthal Report and Baruch Plan, the success of arms control efforts that led towards the 1972 Anti-Ballistic Missile (ABM) Treaty, and episodes of attempted state formation?” (Allan Dafoe’s research agenda)
- “This project explores the impact of US nuclear strategists on nuclear strategy in the early Cold War. What types of experts provided advice on US nuclear strategy? How and in what ways did they affect state policy making on nuclear weapons from 1945 through to the end of the 1950s (and possibly beyond)? How could they have had a larger impact?” (Waqar Zaidi – Some AI Governance Research Ideas)
- “History of existential risk concerns around nanotechnology: How did the community of people worried about nanotech go about communicating this risk, trying to address it, and so on? Are there any obvious mistakes that the AI risk community ought to learn from?; How common was it for people in the futurist community to believe extinction from nanotech was a major near-term risk? If it was common, what led them to believe this? Was the belief reasonable given the available evidence? If not, is it possible that the modern futurist community has made some similar mistakes when thinking about AI?” (Ben Garfinkel – Some AI Governance Research Ideas)
Examples of research exploring historical events to inform AI governance:
- Jade Leung, Who will govern artificial intelligence? Learning from the history of strategic politics in emerging technologies
- Cihon, Maas, and Kemp, Should Artificial Intelligence Governance be Centralised?: Design Lessons from History
- Zaidi and Dafoe, International Control of Powerful Technology: Lessons from the Baruch Plan for Nuclear Weapons
- Katja Grace, The Asilomar Conference: A Case Study in Risk Mitigation
- Preliminary survey of prescient actions – AI Impacts
- Katja Grace, Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation
- O’Keefe, How Will National Security Considerations Affect Antitrust Decisions in AI? An Examination of Historical Precedents
- AI Impacts, Historic trends in technological progress
Possible questions include:
- “Will EU regulation diffuse globally via the so-called “Brussels effect” (Bradford, 2020), or will there be a global race to the bottom with regards to minimum safety standards (Askell et al., 2019; Smuha, 2019)?” (Legal Priorities research agenda)
- “How should the scope of AI safety regulations be defined (Schuett, 2019)? Do we need new regulatory instruments (Clark & Hadfield, 2018)? How can compliance be monitored and enforced? Is there a need for stronger forms of supervision (Bostrom, 2019; Garfinkel, 2018)? If so, would they violate civil rights and liberties?” (Legal Priorities research agenda)
- “Is there a need to legally restrict certain types of scientific knowledge to prevent malevolent actors from gaining control over potentially dangerous AI technologies (Bostrom, 2017; Ovadya & Whittlestone, 2019; Shevlane & Dafoe, 2020; Whittlestone & Ovadya, 2020)? If so, how could this be done most effectively? To what extent is restricting scientific knowledge consistent with the relevant provisions of constitutional law?” (Legal Priorities research agenda)
See the Legal Priorities research agenda for more context and further questions. A survey of research questions for robust and beneficial AI from the Future of Life Institute also contains many questions law students could explore.
It may be valuable to explore topics such as “How can we avoid a dangerous arms race to develop powerful AI systems? How can the benefits of advanced AI systems be widely distributed? How open should AI research be?”
There are a lot of questions you can draw from Allan Dafoe’s AI governance research agenda. You could explore the potential impact of exacerbated inequality and job displacement on trends such as liberalism, democracy, and globalisation; who should be leading on AI governance; how substantial an advantage China has – as compared with other advanced developed (mostly liberal democratic) countries – in its ability to channel its large economy, collect and share citizen data, and exclude competitors; and how a dangerous arms race to develop AI could be prevented or ended.
Other potential questions include:
- “Taking into account likely types and applications of autonomous weapons, what are the likely effects on global peace and security?” (80,000 Hours)
- “What plausible paths exist towards limiting or halting the development and/or deployment of autonomous weapons? Is limiting development desirable on the whole? Does it carry too much risk of pushing development underground or toward less socially-responsible parties?” (80,000 Hours)
- “How might AI alter power dynamics among relevant actors in the international arena? (great and rising powers, developed countries, developing countries, corporations, international organizations, militant groups, other non-state actors, decentralized networks and movements, individuals, and others).” (Center for a New American Security)
These are some of the questions suggested in the Effective Altruism Psychology Lab’s research agenda. They suggest you reach out if you’re interested in pursuing research on any of these topics.
- “How do AI researchers think about risks from advanced AGI? How concerned are AI researchers about risks from uncontrolled AGI, and why? How do AI researchers’ views differ from those of other populations? Are they more or less concerned?” (Effective Altruism Psychology Lab)
- “How can we raise awareness of AGI risk among AI researchers? How do we create group dynamics and systems that ensure safe development of AGI?” (Effective Altruism Psychology Lab)
- Seth Baum’s ‘On the promotion of safe and socially beneficial artificial intelligence‘ is one relevant paper.
- “What do people think about digital sentience? Do they believe that digital beings can be sentient? If not, why? Do people morally value or discount (different types of) potential digital beings?” (Effective Altruism Psychology Lab)
- What cognitive biases could undermine efforts to align AGI? Do people underappreciate how training an AGI on one task makes it competent to do another task? Do people succumb to the curse of knowledge when trying to predict AI’s ability to infer how humans want it to behave? (Psychology for Effectively Improving the Future)
- Eliezer Yudkowsky’s ‘Cognitive Biases Potentially Affecting Judgment of Global Risks’ is one relevant paper.
Possible questions include:
- “Given what we are still learning about ourselves and our values, is it possible to anticipate the direction that our values are moving in, or the direction they should move in?” (AI Governance: A Research Agenda – Allan Dafoe)
- “If something like digital people became possible – digital beings that ought to have their own interests considered – who should have the right to create them, and how should voting work? (The default of “anyone can create as many digital people as they want, without limit” doesn’t seem ideal, especially assuming such digital people would vote.)” (Appendices for “Important, actionable research questions for the most important century” – Holden Karnofsky)
- “How should we value various possible long-run outcomes [of the development of AGI] relative to each other?” (Appendices for “Important, actionable research questions for the most important century” – Holden Karnofsky)
For more ideas, see:
- The post ‘Problems in AI alignment that philosophers could potentially contribute to’ and the comment discussion.
- A survey of research questions for robust and beneficial AI – the Future of Life Institute
Possible questions include:
- “How do AI researchers think about risks from advanced AGI? How concerned are AI researchers about risks from uncontrolled AGI, and why? How do AI researchers’ views differ from those of other populations? Are they more or less concerned?” (Effective Altruism Psychology Lab)
- “How can spread norms in favour of careful, robust testing and other safety measures in machine learning? What can we learn from other engineering disciplines with strict standards, such as aerospace engineering?” (Technical AI Safety Research outside of AI)
- “How can we best increase communication and coordination within the AI safety community? What are the major constraints that safety faces on sharing information (in particular ones which other fields don’t face), and how can we overcome them?” (Technical AI Safety Research outside of AI)
- “What capabilities might AI systems one day have, and what would be some possible social consequences? For example, what would happen if an AI system could generate language and images designed to persuade particular sets of people of particular propositions?” (80,000 Hours)
Further resources
- The Case for Taking AI Seriously as a Threat to Humanity – Vox
- AI Safety from First Principles by Richard Ngo gives a case for why the development of AGI might pose an existential threat.
- AI could defeat all of us combined by Holden Karnofsky gives an argument for why AGI ‘only’ as intelligent as humans could pose an existential risk.
- Preventing an AI-related catastrophe – 80,000 Hours
- What could an AI caused catastrophe actually look like? – 80000 Hours
- Benefits and Risks of Artificial Intelligence – The Future of Life Institute
- Preparing for AI: risks and opportunities – Allan Dafoe (video)
Podcasts
- Prof Allan Dafoe on trying to prepare the world for the possibility that AI will destabilise global politics – 80,000 Hours
- Shahar Avin on AI Governance – The Inside View
How can AI governance research have a positive impact?
- AI Governance: Opportunity and Theory of Impact – Allan Dafoe
- A personal take on longtermist AI governance – Luke Muehlhauser
- How Can We See the Impact of AI Strategy Research? – Jade Leung (2020)
- AI Strategy: Pathways to Impact – Ben Garfinkel (2019)
- If you’re interested in working on AI governance, it’s useful to build a basic technical knowledge of AI (e.g. machine learning and deep learning) as well as knowledge of technical AI safety specifically. Technical knowledge will both improve your understanding of the issues AI governance aims to address and help you gain legitimacy in the eyes of decision-makers. We recommend exploring some of the resources listed in our profile on human-aligned artificial intelligence.
- The case for building expertise to work on US AI policy, and how to do it – 80,000 Hours
- Career resources on U.S. AI policy and European AI policy
- Guide to working in AI policy and strategy – 80,000 Hours
- China-related AI safety and governance paths – Career review – 80,000 Hours
- How social science research can inform AI governance – Baobao Zhang (video and transcript)
- Advice from Seth Baum, the director of the Global Catastrophic Risk Institute, for students who want to work on global catastrophic risks.
- Miles Brundage on the world’s desperate need for AI strategists and policy experts – 80,000 Hours podcast
If you’re interested in a programme that isn’t currently accepting applications, you can sign up for our newsletter to hear when it opens:
- The PIBBSS summer research fellowship is for researchers studying complex and intelligent behaviour in natural and social systems, who want to apply their expertise to AI alignment and governance.
- The Center on Long-Term Risk’s summer fellowship is for researchers who want to work on research questions relevant to reducing suffering in the long-term future.
- The CHERI summer research program is for students who want to work on the mitigation of global catastrophic risks.
- AGI Safety Fundamentals (AI Governance track), is a programme held at the University of Cambridge and virtually, for those interested in building their knowledge of this field.
This database of EA-relevant US policy fellowships may be useful for finding further opportunities to gain experience.
- Apply for our coaching and we can connect you with researchers already working in this space, who can help you refine your research ideas.
- Apply for an 80000 hours coaching call for advice and connections.
- Apply to join our community if you’re interested in meeting other students working on this research direction.
- Apply for our database of potential supervisors if you’re looking for formal supervision and take a look at our advice on finding a great supervisor for further ideas.
Our funding database can help you find potential sources of funding if you’re a PhD student interested in this research direction.
Sign up for our Effective Thesis newsletter to hear about opportunities such as funding, internships and research roles.
Other newsletters that are useful for keeping up with AI governance and advancements in AI are:
- Alignment Newsletter – Rohin Shah
- The European AI Newsletter – Charlotte Stix
- ChinAI newsletter – Jeff Ding
- policy.ai newsletter – CSET
- Import, AI – Jack Clark
Contributors
This profile was last updated 5/10/2022. Thanks to Rick Korzekwa, Jenny Xiao and Lennart Heim for helpful feedback. All mistakes remain our own.
Subscribe to the Future Researchers Newsletter
Explore all our recommended research directions
Search for profiles that are tailored specifically to your degree or discipline using the menu below. If you’re searching for thesis topics, or considering a research career which allows you to make a significant positive impact in the world, we advise you to go through these research directions and learn more about those that seem impactful or interesting to you.
If you are interested in a profile that isn’t listed under your discipline, we still encourage you to explore it if you think you could make progress in this direction. You can also explore all our recommended research directions organised by theme.
- All
- Agricultural science
- Basic medicine
- Biological Sciences
- Business
- Chemical sciences
- Computer and information sciences
- Earth and environmental sciences
- Economics
- Engineering
- Health sciences
- History
- Law
- Maths
- Media and communications
- Philosophy and ethics
- Physical sciences
- Political science
- Psychology and cognitive sciences
- Sociology
See here for a visual map of all our research directions