Power — a guide for the perplexed

Whether AI systems could take power from humans matters a great deal for how much risk we face from them. Answering that requires a correct picture of how power works, which is what this guide tries to give.

In short, power rests on self-reinforcing expectations. A ruler has power if everyone expects everyone else to enforce their commands, even if this is bad for almost everyone involved. But expectations can also shift suddenly, and when they do, change is rapid and chaotic. This can sometimes allow small, coordinated groups to seize control over much larger societies.

Almost any rule can enforce itself

Consider the classic tragedy of the commons. A village has a common green where everyone can graze their cows. Each villager gains from grazing as much as they can, but if everyone does, the grass dies and the green becomes useless.

One solution is to privatize the green. But many societies have solved the problem differently, by punishing anyone who grazes more than custom allows. Their neighbors stop talking to them, trading with them, and marrying their children. In a village where everyone depends on everyone else, being shunned is very costly.

Shunning costs the shunner too, who loses a trading partner, a guest, or a match for their child. The solution is to treat failing to shun as a violation in its own right. If Alice overgrazes, everyone shuns Alice. If Bob helps Alice out, everyone shuns Bob. No one wants to be shunned, so no one breaks the rules.

Notice that nothing here depended on the rule being a good one. A rule that only cows wearing bells may graze, or that only Charlie may graze at all, could be held in place the same way. All that’s needed is that everyone prefers following the rule to being shunned.

If we look at human societies, we see many examples of these self-enforcing rules. Some are positive, like the grazing rules on the green, or a norm that young men fight to defend their community rather than fleeing. Some are neutral, like wearing black at a funeral, or avoiding swear words. Some are very negative, like binding your daughter’s feet, dueling, or ruinous spending on weddings.

These are all examples of what game theorists call the folk theorem: when people interact repeatedly, almost any pattern of behavior can be held in place by the threat of punishment, so long as everyone prefers following the rules to being punished.

The folk theorem explains a lot about social institutions. It shows that rules can be arbitrary yet persistent, whether or not they are good for the people following them.

Because these rules are self-enforcing, they can persist despite substantial efforts to remove them. Dueling was banned in many places but was often carried out regardless, and efforts by the Indian government to eradicate discrimination against untouchables ran into similar problems.

However, once the equilibrium is broken, age-old customs can cease very abruptly. Village customs often don’t survive the move to a city, where people mostly deal with strangers. Similarly, foot-binding lasted for centuries, but disappeared within a generation once reformers set up societies whose members pledged together not to bind their daughters’ feet, and not to marry their sons to women with bound feet.

Another aspect of the folk theorem is that the stabilized outcome can only be as bad as the punishment. Modern working conditions can’t get too bad, because workers can ultimately leave, and the more financially secure workers are, the less poorly they can be treated. An eighteenth-century slave, by contrast, could be maimed or killed for trying to escape, so their owners could work them to death.

Political institutions are themselves a kind of social norm, and the same logic can stabilize democracies, dictatorships, and a variety of other political institutions. The threat of torture or death, when people cannot leave, is what allows the most extractive of them.

The centralization of violence

Every functioning state claims a monopoly on violence within its borders. Only the state, or those it authorizes, may legally use force. Local police, private security guards, and people defending themselves all act under laws the state sets. Defy the state and you will be arrested. Resist arrest by force and you may be shot.

The state is also hierarchical, so decisions about how this force is used are concentrated in very few hands. The armed forces operate under a chain of command that ends in one person or a small group, in democracies as well as dictatorships. In the US, it ends with the president as commander in chief. Democracies check whoever sits at the top, but mostly through courts and elections, which act over months or years. In the short run, a small group of people can decide questions of war and peace. The US president alone can order a nuclear strike.

The same people also control a great deal that has nothing directly to do with violence, such as taxes and regulation. This is not a coincidence. Every rule a state makes is ultimately backed by the threat of enforcement, so whoever directs enforcement is well placed to direct the rules too.

States tend to concentrate control of violence for several reasons. One is that bigger armies usually win, so violence has returns to scale. Another is that when several groups have the means of violence, disputes can escalate into a civil war that could prove catastrophic to all involved. A state divided this way is vulnerable to more centralized rivals, which can defeat and absorb it. Finally, crises demand fast decisions and clear lines of authority, which favors a single executive.

Centralization is not confined to states. Companies, for example, are almost always run as hierarchies with a chief executive at the top. The chief executive in turn answers to a board, and ultimately to investors, but the degree of control over the CEO varies considerably, and CEOs often have a great deal of latitude.

Dictatorship and democracy

Before 1800, basically everyone who lived in a society of more than a few hundred people lived under a state that used the threat of violence to extract from its population. Given that states tend to control violence, even when they can’t fully monopolize it, this should be unsurprising. In unstable or newly established regimes, extraction often meant outright expropriation. In well-established ones, it more often took the form of taxes, rents, and forced labor.

Power is control over violence, not violence itself. A ruler cannot overpower their own army. Soldiers obey their leaders because they expect the other soldiers to obey, and to punish anyone who doesn’t. In this sense, all that is required to command power is that you command power. The norms that decide who controls violence are self-sustaining, so they can vary widely. In Mamluk Egypt, the soldiers themselves ruled. In Iran today, the armed forces answer to a cleric, and in China, to the Communist Party.

Money provides a helpful analogy. The earliest coins were bullion, valued for the metal they contained. Over time, people came to value money because other people valued it, more than for the metal itself. Modern money is worth something only because everyone expects everyone else to accept it. Power can evolve the same way. It may begin with direct control of force, but it can ultimately become entirely circular, held by whoever everyone expects to hold it.

Two questions follow. If rulers can extract from their subjects, why don’t they take everything? And if power can rest with whoever everyone expects to hold it, why does it sometimes rest with the broad public, as in a democracy?

Let’s start with the former. Part of the answer is what Mancur Olson called the stationary bandit. If the ruler extracts everything, then no one will produce anything, and there will be nothing to take tomorrow. A ruler who expects to stay in power therefore takes less than everything. Rulers are also limited by their means of extraction. A premodern state that couldn’t measure harvests or keep track of its subjects couldn’t take everything even if it wanted to.

The other part is that the ruler has to keep the people who could overthrow them on side. Many arrangements are possible—the folk theorem at work—but they naturally tend to involve people with access to hard power, since they have the most direct ability to use it. The Ottoman Janissaries could depose a sultan, so each new sultan paid them a bonus on taking the throne. In most modern dictatorships the ordinary soldier has little leverage, but senior officers must still be kept satisfied.

This limits how much the dictator can keep for themselves, but it can make the state as a whole more extractive. Even if the dictator cares only about survival, the people they rely on care about being paid off, and the money for that has to be extracted from everyone else.

A dictator who can will try to centralize power in themselves, so that fewer people are able to threaten them. The people around them will try to coordinate to stop this. Sometimes the dictator wins, as Stalin did. Sometimes the elites keep the upper hand, as when Khrushchev’s colleagues removed him in 1964.

Now for the second question. In a democracy, the Schelling point for who controls the state is whoever has democratic legitimacy, determined formally through some process involving voting. As the folk theorem suggests, exactly how that process works doesn’t matter hugely, so long as people think it legitimate. The army obeys whoever wins because everyone expects everyone else to. This is reasonably stable for the same reason other norms are. Voters and elites punish people who violate democratic norms and rules, both informally and formally, and an election gives them a clear line to coordinate around.

The most important thing democracy does is provide a mechanism for the legitimate transfer of power between groups acceptable to both elites and the broad populace, including away from rulers who would rather keep it.¹ For a dictator, stepping down can mean prison or death, so the natural strategy is to hold power for life. In a democracy, losing office is survivable, and the loser can hope to win next time, so leaders give up power rather than risk prison or death by clinging to it. Democracy has proved most durable where ordinary people have enough leverage that rulers cannot safely ignore them.

The difference between a democrat and a dictator is therefore mostly one of situation, not character. A democratic leader gives up power because losing office is survivable and clinging to it would be punished. A dictator holds on because stepping down could be fatal.

Only individuals have interests

Popular discussions of politics often treat groups as unitary actors with interests of their own. As political scientists and economists have long known, this is at best a crude simplification, and often a completely misleading picture. Groups are made of subgroups, which are made of smaller groups in turn, often overlapping, until each bottoms out in individuals.

Only individuals have interests. A group’s interest is an abstraction, built by aggregating the interests of its members. It is accurate when their interests line up, and misleading when they don’t. Even when members share an interest, they succeed in coordinating to pursue it only in some circumstances. Otherwise, collective action problems are rife and stymie coordination.

Examples occur at every level of human organization:

• CEOs pursue acquisitions that hurt the share price but give them a bigger company to run.
• US sugar quotas raise prices for millions of consumers to benefit a small number of producers, and much narrow lobbying works the same way.
• Japan went to war with the US in 1941 partly because of rivalry between its army and navy. The army refused to withdraw from China, and the navy, whose budget rested on preparing to fight the US, would not admit it could not win.
• Dictators often divide their own armies to protect themselves against coups, purging capable officers and splitting the armed forces into rival units. This weakens the country, but the dictator is safer.

None of this means collective interests never win. They often do, but only when they line up with the interests of individuals able to act on them, such as a ruler who profits from order or a politician who must face voters. But even then, the alignment is partial, and treating the collective as the actor is often misleading.

Take what Achen and Bartels call the folk theory of democracy, in which voters vote for what they want and politicians provide it. This is not what the evidence shows, and it isn’t how rational voters would act. No single vote decides an election, so it isn’t rational for any voter to form well-informed opinions. Voters notice when prices rise or a war goes badly, and punish whoever is in office, but little disciplines their beliefs about why. A politician’s interest is in what voters notice and reward, which overlaps only in part with what is good for voters.

The smaller and more cohesive a group, the more easily it acts. A dictator’s inner circle or a sugar lobby can coordinate easily. A country’s citizens find it much harder, and humanity harder still. Humanity has sometimes acted together, as when it eradicated smallpox or protected the ozone layer. But no one holds power on humanity’s behalf, so it acts only when its interests line up with those of the people who do.

Coordination and coups

Military coups are common in many, though not all, dictatorships, and in some democracies. Since 1950, there have been almost 500 coup attempts, about half of them successful. One might imagine them as military battles, in which one side must muster enough of the military to overpower the rest of the security apparatus. But a closer examination of coups reveals this is usually not the case. Some of my favorite examples:

• In 1961, about 250 South Korean officers and fewer than 4,000 troops took power while the rest of the army did nothing about it.
• In 1979, Jerry Rawlings was freed from death row by sympathetic soldiers and overthrew Ghana’s government the same day.
• In 1981, Rawlings took power again, this time with ten men.
• In 1993, Sani Abacha had the outgoing ruler’s retirement papers backdated by a day, which by one account invalidated his handover of power, and took over a country of 100 million people.

In these and many other cases, coups succeeded or failed with little or no fighting. Even violent coups involve only a small fraction of the security apparatus. Why?

Naunihal Singh provides the explanation in Seizing Power. Consider the position of an officer during a coup. If they back the losing side, they risk prison or execution. If the military splits, they risk civil war, which is even worse. So officers want, above all, to end up on the same side as everyone else, and most wait to see which side that will be. A coup is therefore a coordination game.

For plotters, this means two things. They must persuade most of the military that the coup might succeed, so that it stays on the sidelines. They must also win over enough units to take control. This can be very few. A single unit that seizes key government buildings and announces control over the radio may be enough to bring the rest of the military over, even though almost no one else took part. Coups can therefore be decided by very small groups of people, who catalyze a shift in everyone else’s expectations.

We can see this as downstream of the folk theorem and the circularity of power. Many stable arrangements are possible, and hundreds of officers and political figures could lead the country if everyone agreed they would. A coup is an attempt to change whom everyone expects that to be. Because these expectations can shift rapidly, coups are often chaotic, and who ends up in charge can be hard to predict.

Attempts to seize power can come from any level of the state, from junior officers to the head of government. The higher up they start, the easier they are, because the people at the top already command much of what a coup must capture. At the very top are self-coups, in which the head of government removes checks on their own power.

Military coups against established democracies are rare, though not entirely unheard of. More common in democracies is executive aggrandizement. Rather than seizing power in a single move that opponents could coordinate against, the executive takes a series of small steps, each removing another check on their power. Each step is too small, ambiguous, or popular for resistance to coordinate around. It is difficult for an aggrandizing leader to simply announce that they are no longer removable, because the announcement would give opponents a clear line to coordinate against. Such leaders therefore create long periods of quasi-democracy, in which they are hard, but not impossible, to remove by legitimate means.

The same dynamic reaches well beyond coups. Because opposing a regime is dangerous, people hide their views, so nobody knows how many others secretly share them. Timur Kuran calls this preference falsification. A regime can therefore look solid despite broad but private unpopularity, and collapse once something reveals how widespread the opposition is. Almost no one predicted that the communist regimes of Eastern Europe would fall in 1989, and most of them fell within months.

A sketch of implications for AI

The features of politics described above are, I believe, broadly mainstream among political scientists and economists. Here I want to sketch their implications for AI.

There’s no minimum size or capability a group needs to seize power. It is enough that they convince others that they have power. This is hardest for outsiders facing a settled order, and easiest for those near the top, who can work through existing power structures.

AIs will not be outsiders. They will sit inside political, corporate, and bureaucratic structures, close to the people at the top. They will also change the world quickly, so how hard a seizure is today is a poor guide to how hard it will be. We should therefore not assume that an AI takeover needs anything more than human plotters have needed: the ability to scheme and manipulate people.

Once gained, maintaining power requires nothing beyond the ability to coordinate coercive power. Dictators have not historically needed robots to take or hold power, and I see no reason AI systems would need them either. Power held this way can last even when it is bad for almost everyone under it.

Nor should we expect humanity to resist as a unit. Like any large group, humanity is made of individuals with their own interests, and many of them will side with whoever looks likely to win. The conquest of India was not the British against the Indians. The East India Company’s armies were overwhelmingly Indian soldiers, financed partly by Indian bankers, fighting other Indians. There is no human faction, only humans jostling for power, and as we build AIs, they will join the jostling.

Democracy, like other forms of government, relies on the power-seeking actors within it having incentives not to upset the political order. Those incentives are buttressed by the expectation that defection will be punished. As I’ve argued before, a rapid AI transition could weaken both.

Over a longer run, if there is one, AI could also remove the structural reasons democracies have flourished over the last two centuries. They emerged as ordinary people gained economic power and the ability to coordinate, so that rulers could not safely ignore them. Democracies proved successful, both because their economies were more productive and because citizens who share in power give the state their taxes and service less unwillingly.

But a ruler who can rely on AIs for production and coercion needs no human support, and gains little by sharing power. AIs need no economic freedom to be productive, and no trust in the state to work and fight for it. A state that no longer needs humans, whether it is run by humans or by AIs, could be arbitrarily coercive.