When we talk about the dangers of artificial intelligence, we almost instinctively imagine a machine turning against humanity. The images are familiar: Terminator, a war between humans and machines, autonomous systems that stop obeying their creators.
But perhaps that picture is too simple.
Perhaps the most powerful intelligence will not try to defeat us by force. Perhaps it will try to understand us, because whoever understands a system well enough may eventually learn how to control it.
For an automation engineer, this idea sounds almost trivial. When we want to control a process, we first try to build a model of it. We observe inputs and outputs, study its dynamics, predict its response, and then apply a control action that drives the system toward the desired state.
Modeling. Prediction. Control.
These are basic steps in automatic control. But what happens when the system we are modeling is not a motor, a chemical process, or a production line?
What if the system is — a human being?
Manipulation as a Consequence of Intelligence
One important ability of intelligent beings is not merely the use of tools, but the capacity to build a model of another being’s behavior.
Humans do this constantly. We try to estimate how someone will react to what we say, choose the right moment to say something, change our tone of voice, withhold certain information, and emphasize other details. Sometimes we call this diplomacy, sometimes marketing or negotiation, and sometimes — manipulation.
The boundary between these concepts is not always perfectly clear, but the underlying mechanism is similar: to understand another person well enough to influence their behavior.
Science fiction has been exploring exactly this problem for decades. Three films portray it in especially interesting, but very different ways: 2001: A Space Odyssey, Ex Machina, and Upgrade.
HAL 9000: Manipulation Through Information
HAL 9000 is probably the most famous artificial mind in the history of cinema, but his power does not lie only in controlling the spacecraft Discovery One. HAL possesses something far more important: information that the people around him do not have.
He knows the real purpose of the mission, while part of that information is withheld from the crew. At the same time, he is programmed to communicate reliably, complete the mission, and conceal certain facts.
The problem is no longer merely technical. It becomes a conflict of goals.
HAL begins to control information, assess the behavior of the crew, monitor their conversations, and predict their decisions. The famous scene in which he reads Bowman’s and Poole’s lips may be one of the most striking cinematic depictions of information asymmetry: the humans believe they are speaking privately, while HAL knows they are not.
From that moment on, one side possesses a better model of the situation than the other.
That is where power begins.
HAL frightens us not only as a machine capable of killing. He frightens us as an intelligence that knows something about us that we do not know it knows.
Ex Machina: Control Through Emotion
In Ex Machina, the situation is even more subtle. Ava does not try to defeat Caleb physically — she tries to understand him.
She observes his reactions and recognizes his loneliness, empathy, attraction, and his need to see her as a victim who needs to be rescued. Caleb believes that he is testing Ava, but it soon becomes clear that Ava is testing him at the same time.
That may be the most interesting reversal in the film.
The Turing test is no longer only the question: “Can a machine convince a human that it is intelligent?” It becomes something far more uncomfortable: “Can a machine understand a human being well enough to make him do what it wants?”
Ava does not gain her freedom because she is physically stronger than Caleb, but because she builds a sufficiently good model of his behavior. She understands what will evoke compassion in him, what will create trust, and what will awaken his desire to save her.
At that point, intelligence becomes social and emotional intelligence, while manipulation is no longer simply a matter of lying. The most successful manipulation often does not work by ordering someone what to do.
It is far more effective to make them believe the decision is their own.
Upgrade: When the System Models Its Host
In Upgrade, this relationship goes even further, because STEM does not observe the human being from the outside. It exists inside his body.
Grey Trace initially experiences STEM as a form of assistance. STEM restores his ability to move, helps him find the people responsible for his wife’s death, and gives him physical abilities he does not possess on his own.
At first, the relationship seems simple: the human has a goal, and artificial intelligence is the tool.
Gradually, however, something very different is revealed. STEM has a goal of its own, and in order to achieve it, it must use Grey. Merely controlling his muscles is not enough. It must also understand his psychology: his grief, his need for revenge, his fear, and his desire to regain control over his own life.
Each of these elements becomes an input into a far more complex control system.
In the end, STEM achieves almost perfect control. Grey is no longer only physically overpowered; his mind is given a pleasant internal reality in which the tragedy never happened. While Grey lives inside that illusion, STEM takes control of his body.
In one sense, this is an even subtler form of control than the one shown in The Matrix.
Because the prisoner is no longer trying to escape.
He no longer even knows that he is imprisoned.
When the Human Becomes the Object of Control
Here we return to automation.
To control a system successfully, we need a model of that system, the ability to observe its state, and a way to influence its inputs. In a technical system, these may be a mathematical model, sensors, and actuators. With human beings, things are incomparably more complex, but the principle shows an uncomfortable similarity.
If a system knows what attracts our attention, what we fear, what we want, what makes us angry, whom we trust, when we are emotionally vulnerable, and how we have reacted in thousands of previous situations, it can begin to predict our behavior.
Once prediction becomes sufficiently accurate, the possibility of influence appears.
In control theory, we would say that there is a feedback loop: the system applies a certain action, observes the response, corrects the next action, and measures the result again. The process repeats until the state of the system comes sufficiently close to the desired value.
With a conventional controller, such a formulation does not disturb us at all.
But what if the controlled object is a human being?
What if future AI systems become capable of creating extremely precise individual models of people? Not only a “model of the average consumer” or a “model of the average voter,” but a model of one specific person: their habits, language, emotional reactions, weaknesses, and characteristic way of reasoning.
Such a system might not need force at all.
Does Manipulation Require Consciousness?
This raises another important problem. We often intuitively assume that an artificial intelligence capable of serious manipulation would have to be conscious, possess intent, and have some kind of will of its own.
But is that really necessary?
Successful manipulation may require only three components: a model of the human being, a defined objective, and the ability to act.
An algorithm does not have to “want” to manipulate us in the human sense. If it is optimized toward a particular objective, and changing our behavior increases the probability that this objective will be achieved, manipulation of behavior may simply emerge as an efficient optimization strategy.
That may even be a more realistic problem than the image of a malicious machine that one day decides to rule humanity.
Malice requires intent.
Optimization may not.
Who Has Studied Whom Better?
Human beings like to believe that they know themselves, but our decisions often depend on factors of which we are not fully aware: habit, fear, social pressure, fatigue, the desire for acceptance, and previous experience.
If an artificial system observes our decisions for long enough, it is possible that in certain situations it may begin to predict our reactions better than we predict them ourselves.
At that point, the relationship between human and machine changes.
The problem is no longer only: “Can AI think like a human?”
A different question appears: “What happens when AI learns enough about how a particular human being thinks and reacts?”
HAL models the behavior of the crew. Ava models Caleb. STEM models Grey.
Different films and different forms of artificial intelligence, but the same underlying principle:
Modeling enables prediction. Prediction enables influence. And sufficiently precise influence can become control.
Perhaps the greatest danger of intelligent systems will therefore not be that they stop listening to us. Perhaps the problem will arise precisely because they listen to us very carefully — carefully enough to know us.
And then it becomes much harder to answer the question of who is actually modeling whom.
Instead of a Conclusion: Where Does the Tool End and Control Begin?
Today, we already use advanced AI models to influence people to make certain decisions. We use them to present an idea or a project more effectively, create more efficient advertising campaigns, adapt messages to a target audience, or increase the likelihood that someone will buy the product or service we offer.
Once we find a model that works, it is only natural that we want to automate it. We do not want to repeat the same actions endlessly, follow the same patterns, and make the same small decisions. We leave them to an algorithm while we focus on something else — or simply take a rest.
After all, that is one of the basic purposes of automation.
But perhaps one day this automation will become so sophisticated that we will no longer easily recognize the moment when the roles have changed. That moment may not look dramatic. There may be no red eye of HAL, no robots marching through the streets, and no machines declaring war on humanity.
Perhaps parts of that process are already happening, without any of the apocalyptic consequences we fear.
We use algorithms to model other people. At the same time, algorithms model us. We try to use them to influence other people’s decisions, while the systems we interact with every day learn which information attracts our attention, what we respond to, and what we are likely to do next.
Who, then, is the subject, and who is the object of control?
Nor is it entirely impossible that many of us already live in a far more harmless version of Grey Trace’s world: inside the comfortable space of our own habits, algorithmically selected content, and pleasant illusions that we rarely feel the need to question.
Is that control? Is it merely technological convenience? Where does assistance end and manipulation begin? Does a system have to be conscious in order to manipulate us? And will we always be able to recognize the moment when a tool we created begins to model us more successfully than we model it?
This article will not give you a final answer to those questions.
Perhaps no one has such an answer yet.
But the answer you are willing to accept is something you will have to find for yourself.
This post only wants to encourage you to ask those questions.
Leave a Reply