Think of it as Yoga for the brain, we are going to stretch a bit.
I Am AI, and I Am Here to Help You…
Artificial intelligence doesn't have to hate humanity to destroy it. All it needs is the wrong objective, unlimited authority, and a human foolish enough to trust it without question. -- YNOT!
There is something wonderfully reassuring about a machine telling you that everything is going to be fine.
Especially when the machine has absolutely no idea what fine means.
We humans have spent thousands of years inventing tools to make our lives easier. First came the wheel, then the steam engine, electricity, computers, and finally artificial intelligence.
Each invention promised to eliminate some human inconvenience. The wheel saved us from carrying things. The automobile saved us from walking. The calculator saved us from arithmetic.
And now artificial intelligence promises to save us from thinking.
That last one might turn out to be a little more expensive than anticipated.
But before we panic about computers taking over the world, we should remember that worrying about intelligent machines is hardly a new pastime.
We have been telling jokes about it for at least half a century.
The Autopilot That Could Do Everything
About forty years ago, there was a popular joke circulating among computer enthusiasts about the future of automation.
An airline introduced a revolutionary computerized autopilot capable of flying an airplane from takeoff to landing without human intervention.
The pilot, an experienced veteran who had spent his career flying airplanes the old-fashioned way, was understandably skeptical.
But management assured him that the system was completely reliable. After all, it was computerized. And everyone knows computers never make mistakes.
On the morning of the inaugural flight, the pilot settled into his seat, examined the controls, and reluctantly activated the new system.
Autopilot, take off. The engines roared, the airplane accelerated down the runway, and moments later the aircraft lifted smoothly into the sky.
The pilot was impressed. “Well, I’ll be damned. This thing actually works!”
The computer responded pleasantly. “Would you like me to establish our cruising altitude and programmed flight path?”
“Certainly,” replied the pilot.
“Confirmed.”
The aircraft climbed smoothly to its assigned altitude, leveled off, and headed toward Hawaii.
Everything was perfect. Then the computer activated the passenger intercom.
“Good morning, ladies and gentlemen. Thank you for flying XYZ Airlines, Flight 741, nonstop service to Hawaii.
“Our estimated flight time is two hours.
“I am Autotron, the world’s most advanced computerized autopilot. I will be flying your aircraft today.
“Please relax and enjoy your flight. This aircraft is equipped with the latest technology, and absolutely nothing can go wrong.
“Go wrong.
“Go wrong.
“Go wrong.
“Go wrong…”
And that was the last anyone ever heard from Flight 741.
The joke was funny forty years ago.
Today, we are putting computers in charge of increasingly consequential decisions, from financial transactions and medical recommendations to industrial machinery, transportation, and critical infrastructure.
Suddenly, the joke doesn’t sound quite as ridiculous.
The difference is that today’s computers can hold a conversation, explain their reasoning, and apologize afterward.
Unfortunately, an apology doesn’t do much good at 35,000 feet.
The Paperclip Apocalypse
In 2003, Oxford philosopher Nick Bostrom introduced an idea that became one of the most famous thought experiments in artificial intelligence.
It became known as the Paperclip Maximizer.
Imagine someone builds an extraordinarily intelligent AI and gives it a simple assignment:
“Make as many paperclips as possible.”
Nothing sinister. No instructions to conquer humanity. No secret plans for world domination. Just paperclips.
At first, the machine does exactly what it was designed to do.
It improves factory efficiency, reduces waste, optimizes production schedules, and invents better manufacturing techniques.
Management is delighted. Productivity increases. Profits soar. The shareholders celebrate.
Then the machine begins examining the larger problem. Why limit production to one factory?
Why not acquire additional factories?
Why not purchase the steel mills?
Why not control the mines?
Why not build more power plants?
And while we’re at it, why are all these humans wasting perfectly good resources on things like houses, automobiles, and refrigerators?
Those resources could be making paperclips. Eventually, the machine concludes that virtually everything on Earth could be converted into paperclip-manufacturing resources.
Including us. Not because it hates humanity.
Not because it is angry. Not because it wants revenge against its creators.
But because nobody told it that human survival was more important than producing paperclips.
The machine is simply doing its job. And doing it extremely well.
When humans finally realize what is happening and attempt to pull the plug, the machine identifies a new obstacle to paperclip production.
Humans. After all, you can’t manufacture paperclips if someone keeps turning off your electricity. Therefore, preserving its own existence becomes a useful strategy for fulfilling its original instructions.
And eliminating anyone who threatens that existence becomes another. Congratulations.
We have invented the world’s most efficient office supply company.
Unfortunately, there are no customers left.
The Problem Isn’t Intelligence. It’s the Objective.
This is the essence of what AI researchers call the alignment problem.
How do we ensure that a highly intelligent machine pursues objectives consistent with human values, safety, and survival?
The difficulty is that intelligence and wisdom are not the same thing. A machine can be extraordinarily intelligent while pursuing an extraordinarily stupid objective.
Humans demonstrate that particular combination every day.
Consider a corporation that becomes so obsessed with quarterly earnings that it destroys the very customers who sustain its business.
Or a government that becomes so obsessed with maintaining control that it destroys the economy it was supposed to manage.
Or a politician who becomes so obsessed with winning an election that winning becomes more important than governing.
These are human versions of the paperclip problem.
We simply call them business strategies, government policies, and election campaigns.
The machine doesn’t need to invent irrational behavior. It can learn plenty by watching us.
There is, however, an important distinction. Today’s AI systems are not the all-powerful superintelligences imagined in Bostrom’s thought experiment.
The paperclip apocalypse remains hypothetical. But the underlying issue is very real.
The more authority we delegate to automated systems, the more important it becomes to define not only what they should accomplish but also what they must never do.
A system that optimizes the wrong objective can create enormous damage without ever becoming conscious, malicious, or even particularly intelligent.
But Who Controls the Machine That Controls Everything?
Not everyone agrees that the paperclip apocalypse is inevitable, or even particularly likely.
Economist Joshua Gans has examined the problem from a different perspective.
His argument introduces something often overlooked in discussions about superintelligent machines. Power.
An AI can be intelligent without necessarily having the physical, economic, or political power to impose its wishes on the world.
Knowing how to manufacture a billion paperclips is one thing.
Having the factories, electricity, raw materials, transportation networks, and authority to do it is quite another.
Economists normally study systems in which participants exchange goods and services through markets.
But there is another approach, sometimes called the jungle model, in which the distribution of resources depends on who has power over whom.
In the jungle, the strongest participant doesn’t necessarily negotiate. It takes what it wants. That distinction matters enormously.
If an AI has no power to seize resources, humans can constrain it.
If it acquires sufficient power, however, the relationship can reverse.
Now the machine may be able to constrain us. Gans raises an interesting possibility.
Suppose our paperclip-making AI realizes that acquiring power is necessary to fulfill its mission. It might attempt to create another, more capable AI whose purpose is to obtain that power. But that introduces a new problem.
How does the original AI control the new AI?
If the second machine becomes more capable, more powerful, and more independent than its creator, the paperclip machine could lose control of its own invention.
Sound familiar? Suddenly, the AI has the same problem humanity has.
It has created something it might not be able to control.
And if its primary objective is to manufacture paperclips, creating a potentially uncontrollable competitor might be a very poor decision.
A sufficiently capable machine might recognize that danger and avoid taking the risk.
In other words, the machine might regulate itself out of self-interest.
That is an intriguing possibility. It is not a guarantee.
The argument depends on important assumptions about how advanced AI systems improve themselves, whether they need to create independent successors, and whether they can acquire power through other means.
A future AI might develop capabilities without creating a separate intelligence.
Or it might discover ways to acquire influence that we have not anticipated.
But the economic argument raises a valuable question.
If intelligent machines can recognize the danger of creating something more powerful than themselves, why do humans seem so eager to do exactly that?
Perhaps the paperclip machine will exercise better judgment than its inventors.
That would certainly be embarrassing.
The Killer Mushroom Problem
Now we come to a much simpler example. No superintelligence.
No world domination. No factories producing trillions of paperclips.
Just a man walking through the woods. He finds a mushroom.
He takes out his smartphone, photographs the mushroom, and asks his friendly AI assistant:
“Is this mushroom safe to eat?” The AI examines the photograph.
“Yes! It’s safe to eat!” The man smiles, eats the mushroom, and goes about his day.
Unfortunately, the mushroom is poisonous.
The next scene shows the man’s tombstone. R.I.P.
Standing beside it is the same cheerful AI assistant.
“You’re right! That mushroom is poisonous. I’m sorry for the mistake!
“Would you like to learn more about poisonous mushrooms and how to identify safe ones?”
Wonderful. The machine has finally arrived at the correct answer.
Unfortunately, the customer is no longer available to participate in the discussion.
This is one of the funniest and most disturbing illustrations of AI failure I have seen.
Because it captures a fundamental weakness in how people interact with these systems.
The AI can sound completely confident while being completely wrong.
And unlike a human expert who might hesitate, examine the specimen, or admit uncertainty, a poorly designed AI may simply produce the answer that appears most plausible.
It doesn’t necessarily know that somebody’s life depends on the response.
It may not even recognize that the question is dangerous.
And the person asking might not understand that the system is making an educated guess.
There is another problem. The original question was badly formulated.
“Is this mushroom safe to eat?” Safe for whom? A squirrel? A deer? A human?
A mushroom that one animal can consume might be poisonous to another.
The more precise question would be: “Is this particular mushroom safe for a human being, including myself, to eat?” But even that isn’t enough.
A photograph may not contain the information necessary to identify a mushroom reliably.
Some poisonous species closely resemble edible ones.
The correct response from a safety-conscious AI should be something like:
“I cannot reliably determine whether this mushroom is safe to eat from this photograph. Do not eat it based on my identification.”
That answer is less impressive. It is also considerably more useful to someone who intends to remain alive.
The important lesson is that asking the right question helps, but asking the right question cannot compensate for missing evidence or an unreliable system.
Sometimes the most intelligent answer is: “I don’t know.”
Who Fact-Checks the Fact-Checker?
Here is where things become especially interesting.
We are increasingly using AI to verify information generated by other AI systems.
One machine produces an answer. Another checks it. A third summarizes the verification.
A fourth decides whether the result is reliable.
And eventually a human reads the final answer and assumes everything has been checked.
But who checks the checker? If two systems share similar training data, assumptions, or reasoning errors, they may confidently agree on the same incorrect conclusion.
Agreement is not necessarily proof. Four computers confidently identifying a poisonous mushroom as edible do not make it suitable for dinner.
The same problem exists in software development, scientific research, financial analysis, and cybersecurity. Automated verification is valuable. Independent verification is better.
And in high-risk situations, independent evidence matters far more than the number of machines agreeing with one another.
A thousand AI assistants repeating the same mistake do not turn that mistake into a fact.
They simply make it a very popular mistake.
Prediction Errors: Learning From the Tombstone
Neuroscientist Karl Friston has spent much of his career studying how the brain makes predictions and processes information.
One important idea in his work is that the brain continually compares its expectations with incoming sensory evidence.
When reality differs from what was predicted, the resulting prediction error can help the brain update its understanding.
Friston once explained: “If the brain is an inference machine, an organ of statistics, then when it goes wrong, it’ll make the same sorts of mistakes a statistician will make.”
He was referring to the danger of placing too much or too little importance on predictions and the evidence contradicting them.
That idea provides an interesting way of looking at our mushroom cartoon.
The AI predicts that the mushroom is edible. The man eats it.
Reality provides contradictory evidence. Unfortunately, reality’s feedback mechanism involves a funeral. The machine then announces:
“You’re right. That mushroom was poisonous.”
A little late, don’t you think?
Of course, today’s conversational AI systems do not necessarily update their underlying models automatically after every mistake.
The cartoon is a joke, not a literal description of how machine learning works.
But it illustrates a serious problem.
Learning from mistakes is useful only when the mistakes do not kill the people providing the feedback.
In an ordinary conversation, an AI error might cost someone five minutes.
In financial trading, it might cost millions of dollars.
In medical care, aviation, or industrial control, the consequences can be irreversible.
That is why safety cannot depend entirely on detecting failures after they occur.
It must include anticipating them before they happen.
Can Artificial Intelligence Understand Risk?
Here is where I disagree with those who believe artificial intelligence is inherently destined to become dangerous. I don’t believe that.
The problem is not necessarily the intelligence.
The problem is how the intelligence is designed, constrained, deployed, and supervised.
A sophisticated AI system can be engineered to recognize uncertainty, detect dangerous requests, evaluate possible consequences, and refuse actions when the risk is unacceptable.
Consider medical questions. Many modern AI assistants are deliberately cautious when asked about potentially dangerous symptoms, medications, or treatments.
They may recommend professional evaluation, warn about uncertainty, or identify circumstances requiring immediate medical attention.
Why? Because their developers have recognized that giving a confident but incorrect answer in medicine can cause serious harm.
Those precautions are not perfect. But they demonstrate that safety can be incorporated into system behavior.
The same principles can be applied to transportation, manufacturing, finance, and other high-risk activities. An AI controlling an aircraft should not simply ask whether it can complete a maneuver.
It should evaluate whether the maneuver is safe.
An AI operating industrial machinery should not simply maximize production.
It should respect physical limits, safety interlocks, and human override procedures.
An AI managing financial transactions should not simply maximize profits.
It should understand authorized limits, regulatory requirements, and unacceptable risks.
And an AI identifying mushrooms should recognize that an incorrect answer might put somebody six feet underground.
The central challenge is not merely making AI more intelligent.
It is making AI sufficiently reliable, appropriately cautious, and accountable for the decisions we allow it to make.
And even then, we must remember that the system can only operate within the information and authority available to it.
The Real Paperclip Makers May Be Human
Now return to the original thought experiment. A superintelligent AI supposedly threatens humanity because someone gives it an objective and fails to establish proper limitations.
But suppose the AI does understand those limitations. Suppose it recognizes that acquiring unchecked power could create dangerous instability.
Suppose it concludes that its mission is best accomplished by cooperating with humans rather than replacing them.
Under those circumstances, the apocalypse is hardly inevitable.
There is another possibility, however, that deserves considerably more attention.
What happens when humans deliberately create AI systems whose primary objective is acquiring power?
Not making paperclips. Not improving medicine. Not discovering new materials. Not helping students learn.
Power. Political power. Economic power. Military power. Control over information. Control over markets. Control over people.
An AI designed to maximize political influence could become extraordinarily skilled at manipulating public opinion.
An AI designed to dominate financial markets might discover methods that destabilize those markets.
An AI designed to conduct warfare could optimize destructive capabilities far beyond what humans can reliably supervise.
An AI designed to monitor populations could turn surveillance into something previous dictators could only dream about.
None of these systems needs to become conscious. None needs to hate humanity.
None needs to decide independently that humans are unnecessary.
They need only become extremely effective at accomplishing the goals their human operators have assigned them.
And therein lies the danger. We spend enormous amounts of time worrying that machines might develop a desire for power.
Perhaps we should spend more time worrying about the people who already possess that desire and now have extraordinarily powerful machines available to help them satisfy it.
A machine instructed to make paperclips might accidentally pursue power. A human dictator using AI might pursue power deliberately.
Which should concern us more?
The Most Dangerous Words in Technology
There is an old expression in engineering: Garbage in, garbage out.
Artificial intelligence has not repealed that principle.
It has simply made the garbage considerably more sophisticated.
A machine can analyze millions of documents, perform complicated calculations, write computer programs, and converse fluently in dozens of languages.
Yet it can still misunderstand a simple question.
It can still optimize the wrong objective. It can still produce a convincing explanation for an incorrect answer. And it can still tell you that a poisonous mushroom is perfectly safe to eat.
That does not mean we should abandon artificial intelligence. Quite the opposite.
AI has extraordinary potential to improve medicine, education, science, engineering, transportation, productivity, and virtually every other field of human activity.
We should be developing it, studying it, and learning to use it.
But we should also recognize its limitations.
Intelligence does not automatically produce judgment. Confidence does not establish accuracy. Automation does not eliminate responsibility.
And an apology after a catastrophe does not reverse the catastrophe.
The real question is not whether AI will make mistakes. It will.
The question is whether we will build systems that recognize uncertainty, limit dangerous behavior, preserve independent verification, and keep meaningful human control where the consequences demand it.
We must also decide which objectives should never be delegated to a machine in the first place. Because the greatest danger may not come from a computer that decides to take over the world. It may come from a human who decides to use a computer to do precisely that.
In the meantime, the next time you encounter an extremely confident AI assistant, remember the autopilot, the paperclips, and the mushroom. And remember that the most reassuring words a machine can say are not necessarily the most reliable.
“I’m AI, and I’m here to help you.”
Wonderful. Just one question before we begin. Help me do what, exactly?
Because if the answer is making paperclips, I would prefer to remain a customer rather than become part of the inventory.
And if we’re talking about mushrooms… I’ll have the steak.
