Complex Adaptive Systems

In what I believe to have been a prophetic book, The Death of Common Sense, Philip Howard writes about government’s increasing emphasis on “process” to the exclusion of accomplishment. As long as the process is in place, everything’s okay. It matters little whether the process actually accomplishes anything, only that we have a defined, repeatable, legally defensible, bureaucratic approach for addressing something.

I flashed on Howard’s writings as I listened yesterday to President Obama address the failures within our intelligence community. In a nutshell, he perceived the problem to be one of process. If we fix the process, then we’ll solve the problem. Obama is a liberal lawyer and his life is dedicated to process, so it’s not surprising that he, as well as the vast majority of the David Brook’s “educated classes” (see my recent post) rely on process. As a consequence, we’re watching the death of common sense.

In a fascinating post at the blog, Armed and Dangerous, Eric Raymond comments on the failures of Brook’s “educated classes” to control or even understand rapidly changing domestic and international events:

Our “educated classes” cannot bring themselves to come to grips with the fact that fundamentalist Islam has proclaimed war on us. They [the educated classes] have run the economy onto recessionary rocks with overly-clever financial speculation and ham-handed political interventions, and run up a government deficit of a magnitude that has never historically resulted in consequences less disastrous than hyperinflation. And I’m not taking conventional political sides when I say these things; Republicans have been scarcely less guilty than Democrats.

In the first month of a new decade, unemployment among young Americans has cracked 52% and we’re being officially urged to believe that an Islamic suicide bomber trained by Al-Qaeda in Yemen was an “isolated extremist”.

One shakes one’s head in disbelief. Is there anything our “educated classes” can’t fuck up, any reality they won’t deny? Will Collier fails, however to ask the next question: why did they fail?

The obvious and most tempting hypothesis for a libertarian student of history like myself is that the Gramscian damage caught up with them. And I think there’s something to that argument, especially when the President of the U.S. more beloved of those “educated classes” than any other in my lifetime routinely behaves exactly as though he’d been successfully conditioned to believe the hoariest old anti-American tropes in the Soviet propaganda arsenal. And is praised for this by his adoring fans!

I think there’s much more to it than that, though. When I look at the pattern of failures, I am reminded of something I learned from software engineering: planning fails when the complexity of the problem exceeds the capacity of the planners to reason about it. And the complexity of real-world planning problems almost never rises linearly; it tends to go up at least quadratically in the number of independent variables or problem elements.

I think the complexifying financial and political environment of the last few decades has simply outstripped the capacity of our “educated classes”, our cognitive elite, to cope with it. The “wizards” in our financial system couldn’t reason effectively about derivatives risk and oversimplified their way into meltdown; regulators failed to foresee the consequences of requiring a quota of mortgage loans to insolvent minority customers; and politico-military strategists weaned on the relative simplicity of confronting nation-state adversaries thrashed pitifully when required to game against fuzzy coalitions of state and non-state actors.

One of the problems with the educated classes is that they typically have little background in mathematics. They perceive complexity as a linear thing, in which adding one or two additional variables doesn’t much matter. They fail to recognize that where complexity is concerned, we live in a non-linear environment. Each added variable increases complexity exponentially, and as complexity increases, simple bureaucratic “process” solutions will fail to accomplish anything. In fact, they may make matters worse.

Richard Fernandez comments on Raymond’s thesis:

Eric Raymond is very persuasive on the point. The problem is that what is required is agile networks but the bureaucrats are still building pyramids. If he’s right, then the “review” and guidelines issued by President Obama will have only a very small impact on the quality of US intelligence analysis, because if there’s anything the bureaucracy can’t stand, it is a complex adaptive system that can “try lots of adaptive strategies and [let] the successful ones propagate”. The lawyers won’t let them do it. It is anathema to the layers and layers of bureaucracy that have accreted over the years. One of the problems with US intelligence reform has been the tendency to fix intelligence failures by layering yet another process over it.

President Obama is supposed to be a smart guy, so instead of adding still more bureaucratic layers to our intelligence apparatus, he should be able to recognize that to combat new, networked threats, we need a complex adaptive system (CAS), not the old bureaucratic model.

But what is a CAS? Wikipedia provides a definition attributed to John Holland:

A Complex Adaptive System (CAS) is a dynamic network of many agents (which may represent cells, species, individuals, firms, nations) acting in parallel, constantly acting and reacting to what the other agents are doing. The control of a CAS tends to be highly dispersed and decentralized. If there is to be any coherent behavior in the system, it has to arise from competition and cooperation among the agents themselves. The overall behavior of the system is the result of a huge number of decisions made every moment by many individual.

Decisions are decentralized; decisions are made low in any hierarchy that does exist; information is networked, and actions emerge and adapt continuously. A commenter ("wws") at The Belmont Club puts it this way:

The two greatest impediments to the initiation of CAS systems are probably also their two greatest strengths: they defy ideology of any kind, and they defy heirarchy.

Ideology on any level seeks to direct actions towards some wished for outcome in order to achieve some other, ulterior ends, which is what Rahm Emmanuel was talking about when he noted that one should “never let a good crisis go to waste.” But a CAS system searches for any possible solution and pursues it without regard to any overarching ideology. As Al Davis became famous for saying, “Just win, baby!”

And the same thing goes for hierarchy – all the higher levels of a hierarchy have the need to stamp their imprimatur on a course of action just to prove that they are still vital to the effort, and as such a true CAS system is an affront to their existence. For an effective CAS system to be implemented, any hieirarchical command structure over it must be eliminated completely. It must be completely autonomous, and every current stakeholder in the US Federal system will fight that tooth and nail.

Barack Obama wants to be an agent of change. So be it. Instead of adding still more (ineffective and counterproductive) process layers to things like healthcare or intelligence gathering and analysis, he might consider a different approach. Create a competitive shadow CAS environment in which these important issues are addressed outside the conventional process meme. The results might be surprising.