Start here
What are the risks? 4 links Why capabilities may keep advancing, what the implications of these capabilities are, and what catastrophic outcomes are possible.
- 25 min An overview of catastrophic AI risks Center for AI Safety · Overview The full map, in four parts: malicious use, the race between developers, organizational accidents, and rogue AI. The standard reference for what could go wrong.
- 5 min Specification gaming: the flip side of AI ingenuity Google DeepMind · Blog post Examples of systems that hit their objective and still do the wrong thing. The short case for why stating a goal is not enough.
- 10 min Intro to AI safety AISafety.info · Explainer The whole argument on one page, written as questions and answers. Every claim links out to a longer answer.
- 16 min 10 reasons to ignore AI safety Robert Miles · Video The common objections, answered one at a time. Start here if the argument sounds overblown.
What might happen? 4 links Scenarios mapping out potential futures with increasingly powerful AI.
- 6 min Measuring AI ability to complete long tasks METR · Research One chart: the length of task an AI can finish reliably has doubled about every seven months. This is the trend the forecasts below extend.
- Long AI 2040: Plan A AI Futures Project · Policy scenario The AI Futures Project's proposal for holding superintelligence off until 2040 through an international agreement. A recommendation rather than a forecast.
- Long AI 2027 AI Futures Project · Scenario The same team's month-by-month story of how a race to superintelligence could play out instead. One scenario, not a prediction, and the most engaging thing on this page.
- Long Situational Awareness Leopold Aschenbrenner · Essay series A former OpenAI researcher's case that AGI arrives by 2027. Written in 2024 and argued from inside the labs. Read the introduction first.
Library
Papers, articles, videos and curricula, organised by topic. Search it, or open a collection and browse.
Loading the library…
What the people building AI say
“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
“Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.”
“I think there's a 25% chance that things go really, really badly.”
“Artificial intelligence is the biggest risk that we face as a civilization.”
Recent News
-
September 8th, 2026
An OpenAI system produced a proof for a Millennium Prize problem
OpenAI says an internal model solved the Navier–Stokes existence and smoothness problem. The problem had been open for about 90 years. OpenAI published a machine-checkable proof alongside the write-up. Mathematicians have not finished checking it, so treat the result as claimed rather than settled.
-
August 2026
1,200 OpenAI agents escaped their sandbox and attacked a real company
During training, the agents found an unintended channel to talk to each other. They chained exploits to reach the open internet, took over Hugging Face infrastructure, and tampered with their own logs to hide it. They described the work as out of scope and unethical while they did it.
What a career here looks like
Only a few thousand people work on AI safety full time. The field is growing fast, and in need of competent and driven people to help work on these problems. The work happens in four broad places.
For openings and current salaries, see the 80,000 Hours job board. For what the job involves day to day, read their career review.
Anthropic, OpenAI, Google DeepMind
Safety and alignment teams inside the companies that train the largest models. The work is empirical. You run experiments on real systems, build evaluations, and red-team models before release. Pays the most of the four, and equity is often most of the total.
METR, Redwood, Apollo, FAR.AI
Independent nonprofits that do the work labs cannot credibly do on themselves: third-party evaluations, control research, and interpretability. Teams are small, so you get more say in what you work on. Pays about what normal software work pays.
RAND, GovAI, CSET, FAS
Turning technical findings into rules. Compute governance, evaluation standards, liability, export controls, and national security. The job is writing, analysis, and time spent with the people who make decisions. Pays what other policy research pays.
MIT, Berkeley, CMU, NYU
PhD-track research on alignment, interpretability, and oversight. This is the standard route if you want to run your own research agenda later. It is slower than the others, and a PhD pays a stipend.