Architects - Masters of Order and Unorder?

In "The new dynamics of strategy: Sense-making in a complex and complicated world" Cynthia Kurtz and Dave Snowden introduce the term "unorder". As "undead" describes something neither truly dead nor alive, "unorder" describes things which are neither systematically ordered, nor completely disordered. In chaotic or complex real-world systems patterns of behaviour or structure can often be found. A system showing these "emergent" patterns exhibits unorder.

Systems can be both ordered and unordered at the same time. For example an organisation can have a formal, ordered management hierarchy, but its decisions are driven, at least in part, by an informal influencing network.

The architect's traditional domain is ordered. He or she receives a coherent set of requirements from identified stakeholders, and uses his technical skill to derive a solution structure which optimally balances the various design forces. Lucky architects occasionally get to work in this deterministic fashion.

But in the real world, it doesn't often work like that. The stakeholders may be difficult to identify, or keep changing. The requirements change frequently, and conflict with one another or the overall objectives. Architecturally sound solutions are derailed by politics, or because the technology doesn't work as it's supposed to. These are characteristic of an unordered domain.

However, the architect's skills may be of great use in such circumstances, proposing structure to help stabilise the chaotic, or using analysis and abstraction to look for patterns in complexity. Having identified emerging patterns, he can help suggest or take action to reinforce and stabilise desirable patterns while discouraging undesirable ones.

The Cynefin Framework

In their paper, Kurtz and Snowden introduce the Cynefin Framework, which divides the space in which we make decisions or solve problems into four "domains":

In "known" space, things are nice and predictable. We know exactly how to handle things, with established, deterministic rules and procedures, and predict the outcome. For each new problem, we simply have to categorise the data, and apply the appropriate rule. This is the domain in which "best practice" has most meaning.

In "knowable" space, things require a bit more work. We may not know the answer immediately, but know that someone does, or that an answer can be found through formal analysis. The solution may be complicated, but it won't be "complex" in the mathematical sense - we can establish clear cause and effect relationships and act on them.

"Complex" situations are not so amenable to analysis. While cause and effect relationships exist, they are neither simple nor direct, and patterns of behaviour may vary over time, so you can't guarantee repeatable outcomes. Formal analytical processes will run into several problems (see the next section for details). We can probe the situation, look for emerging patterns, and work to stabilise and exploit them, but don't expect everything to boil down to simple, stable rules.

Finally, we have the domain of chaos, where behaviour seems random, unrelated to causes. The only thing we can do here is try and help bring some stability. Of course, no real situation is truly random, but a "very complex" state may be the same in effect.

The known and knowable are the domains of order, while the complex and chaotic domains are those of unorder.

The Kurtz and Snowden paper also considers transitions between the domains. Situations change over time. As our knowledge increases, there's a natural progression clockwise from Chaos to the Known, while forces of unorder tend to push systems the opposite way. Transitions can be gradual or sudden, and can be decisively one-way, or an oscillation between two domains. The following picture identifies the typical transitions with some explanatory labels:

Matching Solutions to Your Domain

The Cynefin framework helps you to understand why different solutions are appropriate in different circumstances. If you can characterise your problem space using the model, it may help direct you both in how you should act, and the type of solution you should be designing.

Structured, analytical methods suit the known and knowable domains. Both packages and bespoke solutions may be suitable in either domain, but it is more likely that the known will be supported by standard solutions, possibly with minimal analysis, while the knowable requires careful analysis and may require unique solutions.

In complex or chaotic situations probing, agile, techniques are more appropriate. Attempting to impose formal, "big analysis up front" methods will fail, as the analysis will encounter a variety of problems:

Things may be changing so fast the analysts simply can't keep up,
Formal analytical techniques may be overwhelmed by detail, or too many variables,
Repeated attempts at analysis may get a different answer every time.

If any of these happens then "analysis paralysis" will set in, and any formal "waterfall" process is set to fail.

In chaotic situations change is the dominant characteristic, and very lightweight, reactive developments the most appropriate solution. The introduction of some solution, even a package or an agile prototype, may itself help to bring stability, but sometimes the most stabilising thing is to discourage any change, and a good architect should recognise when that's the right approach.

Solutions in the known space are often relatively inexpensive and painless, so management tends to seek them out. The difficulty is that they won't fit problems in the other domains. Part of an architect's value is in recognising this mismatch, and explaining to management why "buy a package" is the wrong answer.

Similarly a lot of IT procurement assumes a known domain: "We're building an X, so we want a ready-made expert in X". But if the problem is in the knowable or complex domains, you really want an architect who has skills in those domains. And that may mean someone who doesn't know it all already, but is willing to learn...

If you're designing a solution, you need to understand the "dimensions of change", so your solution has appropriate flexibility. Thinking about what is genuinely known, knowable or unknown about the problem may help. Look not only at the known requirements, but at any historic pattern of change or instability in them. (See my paper "Strategies for Flexibility" for more ideas.)

If your problem is inherently complex then you may need a system which seeks the best answer itself. Evolutionary, "genetic", algorithms or a "Monte Carlo"-type simulation may be better than an analytical solution. You may also need to guide against a simplistic imposition of order, by making sure your models are of sufficient complexity.

In unordered space traditional roles and responsibilities may be blurred. An architect may need to cross the boundaries with management, analysis and development roles, acting as needed to help find the right answer. Your own agility of approach may be an important contribution.

The Forces of Unorder

The architect must understand the forces of unorder. For example, rational analysis and design is founded on three questionable assumptions:

People act deliberately. While usually true, we have all done things by accident, on a whim, or on a hunch. So does everyone else.
People act consistently. This is untrue. A human's decisions and actions are guided by context. An individual may act differently for himself, for his family, for his company and so on.
People act to maximise reward. Again untrue. Altruism aside, humans are good at considering the big picture as well as local forces, intangible as well as tangible benefits. Simplistic models which don't consider these will produce the wrong result.

The architect must question requirements, constraints and technical assessments which don't seem to fit an appropriate pattern. Failure of one or more of these assumptions is usually the cause, and if you understand the reason for someone's position, then it's much easier to either work with or work to change it.

Finding Patterns and Using Models

We use patterns and models to describe behaviour, problems and solutions. In ordered space it should be possible to apply existing analysis and solution patterns. In unordered space the patterns may be new, and they may emerge from apparent chaos, but finding them could be a major contribution to progress.

Architects and analysts will naturally look for "technical" patterns in data, business rules or solution components. However, they must also be aware of patterns of a commercial or political nature. In unordered space these will strongly influence rates of change, the emergence of other patterns, and the acceptability of possible solutions, more so than in ordered situations. In the example of a company with both formal and informal influencing networks, smokers may have undue influence on a company's decision making, because they have the ear of the CEO on his cigarette breaks.

Emergent patterns may be positive, for example helping to understand the behaviour of a complex system. But they can just as easily be negative. For example gradually decreasing system performance may indicate an incipient failure. The architect must be alert to patterns of all forms, seeking a thorough, clear explanation whether or not it brings good news.

Abstraction is a vital tool in all of this. Separating the general from the specific is key to finding emergent patterns, and also to designing solutions which manage complexity and rapid change. Focus not only on what's different, but also on what's the same, but always be on your guard against imposing simple solutions where they don't belong.

As patterns emerge or models develop, they must be explained to appropriate stakeholders. It is always one of the architect's duties to help do this, but it becomes essential in complex situations. Stories which use metaphors or historical anecdotes can be a very powerful way of communicating complicated or technical issues using everyday language. Story techniques are becoming a popular tool for this reason. The Cynefin website contains a number of interesting papers about them.

Alternatively, clear, simple pictorial models may help. These may be the best way to explain the degree of complexity, or forces acting to push a system from an ordered state into an unordered one. Clear, concise visions can help reinforce the development of desirable patterns, and support moves from complexity to order. Provided they are not abused, tools like Strategic System Visions can promote a "virtuous circle" of both stability and increased understanding.

Conclusions

IT Architects or their clients who work only in ordered space are either extremely lucky, or deluding themselves. Most of us work in environments with significant elements of unorder. A wise architect works to understand the nature of his or her domain, exploits appropriate architectural skills and tries to develop well-matched solutions. I hope that the framework and ideas presented in this paper will help you to do just that.

References

The Cynefin Framework is described in "The new dynamics of strategy: Sense-making in a complex and complicated world" by Cynthia Kurtz and Dave Snowden, originally published in the IBM Systems Journal Vol 42 No 3, 2003. (Also available in pdf).

"Cynefin" is a Welsh word analogous to "heritage" or "background" but with a wider scope, essentially "everything which makes us what we are". The Cynefin web site is at www.cynefin.net.

Organisations need to protect and maximise the value of their IT assets. To protect against threats from business and technological change systems need to be flexible: able to change to support new functions, new workloads and new working environments. Flexibility does not happen by accident - it is usually the result of planning, forward thinking and adopting strategies known to enhance and encourage it. My paper "Strategies for Flexibility" (PDF format) presents some of those strategies.

A number of books discuss the issue of situations too complex or too chaotic for normal analysis, but one book focuses on these issues specifically. It's Wicked Problems, Righteous Solutions, by Peter DeGrace and Leslie Hulet Stahl, (Prentice Hall 1990). You can buy it from Amazon UK or Amazon.com.

While not focused exclusively on these problems, AntiPatterns, by William J. Brown, Raphael C. Malveau, Hays W. “Skip” McCormick III and Thomas J. Mowbray (Wiley 1998) is an amusing book about common ways in which things can go wrong, and how to recognise when they do. If you think you’re failing, and you’re not sure why, this is a good place to start. You can buy it from Amazon UK or Amazon.com.

Comments

If you'd like to comment on this article, with ideas, examples, or just to praise it to the skies then I'd love to hear from you. Please click here.

Richard Veryard on 25/2/05

Are there specific situations in your architecture practice that call for a top-left approach? I think it would really progress the debate and the understanding if we had some solid examples of how the Cynefin framework applies to specific architectural projects in specific businesses. Cynefin is currently being picked up by the Agile Methods crowd, and it would be good to have some practical examples of how Cynefin makes a real difference to what architects can achieve.

My Reply

Yes, I do have some real, current examples where complexity is forcing me to say to the client "you can't analyse this". Watch out for a follow-on "examples" piece sometime soon.