Imagine we continue to develop more and more powerful AIs to write our emails, search the internet, and change our calendars. As our science improves, the AI’s start showing signs of a theory of mind, lying, and even tool use (Oh darn GPT4 already does). The companies making the AI’s are making lots and lots of money. Everyone who can starts developing Artificial Intelligence as to make money. An arms race begins in this new intelligence gold rush.

But then BAM, someone, somewhere steps on the metaphorical landmine. Someone creates an AI that starts exponentially self-improving itself.

It's weird for us, humans, to consider but imagine you could easily self-improve. I’m not talking about slowly learning some language, I am talking about adding more space inside your head and downloading the language. I’m talking about bulking up your memory, your attention span, and your rate of processing information. I am talking about manipulating how many times you loop over a problem and increasing the fidelity of your imagination.

At a certain point, AI may gain these abilities of self-improvement. And the thing about self-improvement is that it is exponential, with no known upper limit. Self-improving helps you further self-improve on and on, exponentially, and we don’t know when that process would ever stop. It’s an intelligence explosion.

This is called a FOOM event. Fast take-Off Machine intelligence. If this happens… we really really want that fast take off AI to be aligned.

Kill all humans?

Will a super artificial intelligence with an unaligned goal kill us all? Well let’s get out the pro and con sheet.

For the pros. If the AI kills all humans, it removes its main competition. The AI removes humans obviously, but more importantly, the AI also removes the possibility of new artificial intelligences arising. The most dangerous thing to a super AI is another super AI. With no competition, an AI can go about completing its goals whatever they may be.

On the Cons side, we have three main points.

First, there may be consequences from unknown powers. Aliens, other unknown AGI, Gods, beings outside of the simulation, etc. The AI may take this possibility seriously. Afterall, we don’t know if there are aliens truly watching us, we don’t know if we are inside a simulation. If one AGI exists, there may already be another, etc. Any of these unknown powers may want to destroy such an obviously uncooperative AI.

Second, the AI’s internal makeup and goals may preclude it from thinking about killing all humans… whether this was designed to be or not. The AI may have weird kinks, for example, the speed at which the AI runs may make it quite short run driven. Perhaps we are dealing with a very impatient super-intelligence. Killing all of humanity, and then rebuilding an entirely self-aligned economy towards the AI’s one goal… will take time. If you are experiencing time 1000 times as fast, perhaps that delay is simply too much. It would be much faster to utilize humanities already created economy.

Third, the hippie argument. Killing all humans and other possible AGI’s reduces cooperators. Perhaps cooperators are good. Having multiple different semi-aligned entities cooperating may create a world where the AI’s goal can be better achieved. Afterall humans as a group are much more powerful and can accomplish much more than a single individual… perhaps the same is true of AGIs.

Will this happen?

We don’t know if Fast take off machine intelligence is likely. First, we do not know how many technological barriers separate us from a FOOM event. Second, we don’t know whether we will develop methods of controlling AI such that FOOM events become unlikely. Third, we don’t know if multiple agents evolving together, will preclude one single agent from taking over.

Regardless I think we as a species should be taking the possibility seriously and be discussing counter measures.