If AI really is risky, then opening it up could be a major mistake.
I’m not quite sure how humanity survived the advent of nuclear weapons without destroying itself — so far — but one thing that likely helped was the simple reason that it’s very hard to build a nuclear bomb. It requires refining uranium, which can’t be casually done in a basement or even in a secret government project. It requires overcoming half a dozen technical hurdles, which requires time and the resources that only a state can gather.
As a result, only nine countries have nuclear weapons, and efforts to reduce nuclear weapons are largely carried out through negotiations among a small number of actors, which have at least some ability to hold to and enforce treaties.
