Managing for Serendipity or why we should lay off “best practice” in km david j snowden

Download 31.7 Kb.
Date conversion07.05.2017
Size31.7 Kb.
Managing for Serendipity

or why we should lay off “best practice” in KM

David J Snowden

Cynefin Centre for Organisational Complexity

IBM Global Services

Over the last two to three years I have asked well over 100 audiences at conferences, in company workshops and academic seminars a simple question: “What spreads fastest in your organisation, stories of failure or stories of success?” The inevitable answer is failure and there are good reasons for this. Over the millennia the human race has come to realise that being equipped with several stories of failure is far more valuable than a story of success. This implies that the common knowledge management focus on best practice is in effect contrary to natural practice; an attempt to impose an idealistic structured process onto the natural activity of learning and knowledge transfer through a focus on efficiency at the cost of effectiveness.

The adoption of best practice implies that:

  1. there is a best way to do something,

  2. we can identify and codify what that thing is,

  3. we can then get employees to follow best practice,

  4. and that it is desirable that they should do so.

In this article I want to argue that in other than a very limited set of circumstances all four statements are false and that in fact best practice is simply entrained past practice. I will start by establishing some basic principles relating to human decision making to provide a framework to examine the above questions. I will then conclude that a major area of knowledge management practice should be to create worse practice systems on the grounds that they provide better and more resilient approaches to learning.

Human decision making: patterns and context

If, and it is a very big if, there is a stable and repeating relationship between cause and effect in a common context then best practice can and should be mandated, Human social systems are uniquely able to create such stable contexts by agreeing and establishing conventions for matters such as payment systems and traffic regulations. In the pantheon of management consultancy techniques this is the domain of business processing re-engineering and its use, together with the growing power of technology enables highly efficient systems which save costs, improve reliability etc. etc. Process re-engineering and legitimate best practice both rely on stable repeating relationships with information or appropriate materials available for each decision point, such decisions being rule based (if X and Y but not Z then take action A). The model is a cybernetic one and cybernetic models are de facto applied to human decision making where humans exist in the process.

Here is where things go wrong for two reasons.

Firstly, humans do not make rational logical decisions based on information input, instead they pattern match with either their own experience, or collective experience expressed as stories. It isn’t even a best fit pattern match but a first fit pattern match (Klein 1998). The human brain is also subject to habituation, things that we do frequently create habitual patterns which both enable rapid decision making, but also entrain behaviour in such a manner that we literally do not see things that fail to match the patterns of our expectations.

Secondly, not all systems are ordered in the sense of repeating and empirically verifiable relationships between cause and effect. In complex systems patterns emerge as a result of multiple interactions between agents and only repeat by accident; they are coherent in retrospect but not in advance. It is easy to be right with the benefit of hindsight, but to define best practice on the basis of past events in a complex system represents folly; especially as most matters relating to human and market behaviour are complex not ordered. As understanding of complex systems is of increasingly importance to knowledge management (Snowden 2002)

Left to their own devices humans are remarkably good at dealing with this lack of “order”, indeed pattern recognition, the ability to manage patterns and our ability to store knowledge in the external scaffolding (Clark 1997) that humans erect around their social systems is at the heart of human intelligence. We use social networks and various other clues to guide our future behaviour; we do not work on the rule based approach of computers.

One of the basic validation techniques used by humans is to create the conditions for serendipity. Again a question asked to the same conference audiences. Given a difficult project, one of those you couldn’t get anyone else to volunteer for, do you drawn down best practice from the Organisation’s knowledge management system or do you go and find five or six people you trust/respect and ask their advice, hear their stories? The answer is not always 100% for the stories, there is an occasional Knowledge Management professional with a vested interest in arguing the benefits of their creation but the (admittedly anecdotal) evidence is overwhelmingly in favour of the stories. We actively seek out multiple encounters to increase the probability of an emergent solution, that does not just repeat the past, but which opens up new possibilities.

We also work very strongly on the basis of using shared context to determine confidence in future actions. A third, but this time rhetorical question to my conference audiences will illustrate this. The audience are asked to imagine three scenarios:

Firstly a person you have known for years and worked with on many occasions, through both good and bad, phones you up to ask a question. You know and trust the person and there is no inhibition to answering. You know what they mean by the question, you know how they will understand the answer.

Secondly another member of the organisation phones you up and asks the same question. You have no prior knowledge of this person and no experience of working together; you have no shared context. Your first task is to create a context, you ask a question, respond to the answer, compare experiences and at the end you share knowledge, but the sharing is inhibited: “Start like this then phone me up again”; “If any of the following happens contact me straight away”; “Why don’t I come and get you started?”

Thirdly some total idiot with the title Chief Knowledge Officer comes along and asks you to write down what you know without any context.

The point is a very simple one, shared context is vital to knowledge exchange, and such context always involves some human trusted validation. This is not to say that codification of material in advance of need is not advantageous, but the effective reference is nearly always human. We do use written material, it represents reflective knowledge and has value, but we normally check out what is or isn’t relevant within a trusted network. An interesting phenomenon is the use of best practice databases not for their material, but to find the human authors of those documents and then make a person to person contact; a lot of money spent on an expertise location system for which there are better solutions. A computer will utilise information and rules that are place before it without question, humans are savvier and knowledge management practice needs to relate to their needs. There is an interesting sidebar here – ask someone a direct question in the context of a real need and social obligation normally results in a voluntary act of knowledge sharing, outside of that context is more likely to result in a negative outcome.

The process of codification of knowledge is a process of abstraction, as we rise through successive levels of abstraction we can have richer and richer conversations with fewer and fewer people. This is understood in the context of expert language by most knowledge management practitioners, but what is often neglected is that for humans, abstraction (and therefore shared context) includes common past experiences, beliefs and values. These common assumptions are rarely stated, because they are mostly taken for granted. One of the related problems with best practice is that when people communicate they often forget the degree to which they have relied on shared common experience of which they are only partially aware. The all too frequent response is “I thought you knew that?” or some variation on the theme.

Is there a best way to do something?

In an ordered system a “best way” is theoretically possible as we are dealing with repeating relationships between cause and effect. If we are dealing with a complex system then there is no such repetition. Even in an ordered system the degree to which we understand the relationship between cause and effect determines the degree to which we can define best practice. This is true even of scientific knowledge where serendipity is as frequently the cause of major breakthroughs as is disciplined method and where old knowledge frequently used best practice to exclude new thinking. My favourite example of this latter tendency is the Longitude story in which the clockmaker Harrison is ignored by the scientists of the day for over a decade because they were convinced that measuring the distance between the moon and the earth was “best practice” and attempting to create a clock that kept accurate time on ship board an illegitimate approach from someone who was not a real scientist. The Longitude story is repeated all too frequently in the day to day life of organisations. For complex systems best practice is dangerous, for ordered systems it is valid, but not universally and only in very stable situations, in all other cases it is entrained past practice.

Can we codify knowledge?

Now let us assume that there is a situation in which there is a right way to do things, a way that is the right way more than once and which can be discovered. The next question is can be codify it in such a way that someone else can pick it up and use it? I am using codify here in the sense of writing things down as this is the most common approach to best practice. One of the basic rules of knowledge management is that we always know more than we can say and we will always say more than we can write down. The loss of content, but particularly context involved in codification means that written knowledge is only ever a partial representation of what we know. There is value in codification provided we do not assume complete capture. Time pressures on staff mean that even where they can codify they are often only able to do a partial job, it is also true that human knowledge is deeply contextual, it is triggered by circumstance, if the author of a document was not properly stimulated at the time of the codification they not remember all of the circumstances that should qualify the application of best practice. The more expert the person doing the codification, the more they will take for granted in respect of their audience, and the more danger there is in following the content of the document without access to the expert’s understanding of context.

Will people follow best practice

I remember when I was in primary school and a nine year old from the next class up was sent to read his essay to each class in school. The headmaster had decided that the essay, a fox hunt from the fox’s perspective was a role model that we should all follow. The essay produced several reactions. The sycophants in the class all proceeded to write essays about fox hunts from the fox’s perspective, those subject to the tyranny of the green eyed god speculated that his mother had written it and a small group of unmentionables took him round the back of the bike shed for the treatment normally accorded to teacher’s pets. There is very little difference between the average eight year old and most employees in respect of their appreciation of something held up to them as best practice. For some people they know what really went on, or think they do and feel that essential facts have been left out. Others resent the fact that key aspects of work that they did have been left out. Now, if someone I respect and trust does something or recommends something then it will achieve results, but that level of trust will never transfer to a “system”.

It is also true that habituation is necessary for the consistent application of best practice. Fire fighters do not just enter each situation with a manual, they practice daily to ensure that best practice is engrained in their thinking, and that practical experience provides both knowledge of when not to follow best practice, and also creates high levels of trust based on interdependency (Weick & Sutcliffe 2001).

This has implications for much of the so called attempts to create efficiencies in human actions. A large part of the attempts to introduce process improvements in professional services for example fails to recognise this need for habituation. Removing administrative and secretarial staff from professional staff on the grounds that these tasks can be carried out by the professionals themselves impacts badly on productivity. At first sight it looks easy; we define a process for completing say timesheets based on best practice, put in place computer support and then lay off the secretarial staff. But real practice is not best practice; some account codes are not entered in time, detailed instructions that an administrative assistance would internalise as they work across the full range of activities have to be looked up afresh each time by the professional and each unit task takes longer and builds frustration. For a computer there would not be an issue as each task would look up the processes on the basis of articulated decision rules, but humans do not work that way, they need to build and habituate patterns to be effective.

There is also a major question as to transferability of best practice. Weick & Sutcliffe (op cit) argue that there are lessons from the behaviour of fire fighting crews, air craft carriers and the like relating to openness to failure than can be applied in industry. This is idealistic to say the least. The context that creates the need for failure sharing in a crew of fire fighters is not common in organisations. In a crisis all organisations tend to increase levels of trust, it’s a human reaction, but to have an organisation maintain that level on a constant basis then they would be constantly lighting fires. Context is the be all and end all of knowledge management.

Should people follow best practice?

Even if we can define best practice, and assuming we can mandate and ensure conformance there remains the question as to the desirability of such conformity. To return to my childhood experience of the essay written by a nine year old, one reaction was to imitate the essay, rather than to use it as an example to stimulate original writing. The worst are those who follow best practice uncritically on the grounds that they cannot get fired for doing so. In one project I ran some years ago, removing artificial intelligence in a computer based best practice system enabled experts to apply their knowledge. A previous project had sought to capture expert knowledge and codify it into a system. The net result was that calling “gut feel” when it went against the computer recommendation was dangerous to your career, while following the computer recommendation meant that there was someone else to blame.

It is also the case than any explicit practice can be used against us. In one lessons learnt programme looking at major systems sales for an organisation we found a case of worst practice, the team had signed up to a contract that was loosing the organisation $10m a year with no escape clause. They had only been kept on to allow us to study what had happened; the plan was to fire them for understandable reasons. However at the end we ended up recommending keeping them on and together and deploying them on the next major bid. The reason for the fault is that the buyer was a former employee of the organisation; he had been on the same training courses as the people who were selling to him. He knew just how to behave to trigger best practice responses based on him being the “champion”. It’s a long and elaborate story, but over the course of six months he gradually sucked them into a relationship from which they could not escape. However when we compared the failed team with another held up as the most successful it was evident that the learning achieved by the failed team was more profound. The successful team were arrogant and over confident.

An alternative to best practice

Firstly I should make something very clear; there is a legitimate and valid domain for best practice. I want no ambiguity or active learning in respect of internet payments or in safety procedures in a nuclear power plant. Best practice is an important knowledge management function; it requires discipline, time and resource. We simply cannot afford the costs of base practice for other than a limited number of cases. It’s rather like mission critical software development where two teams work in parallel on the same code and the results are compared. It’s expensive, but for say an air traffic management system it is justified.

However the range of circumstances in which we can really afford to invest in best practice is limited, even when it is appropriate, so we need to turn to other tools and techniques. Its worth remembering that the primary purpose of knowledge management is to enable better decision making and to create the conditions for innovation; better decision making is contingent on active learning, innovation is dependent on disruption of entrained patterns of thinking. In this final section I want to look briefly at some of them, reflecting on current research and experimental consultancy within the Cynefin Centre.

Narrative Databases

We normally learn by hearing stories from diverse sources, synthesising the learning with our current situation and determining a plan for action. Properly constructed narrative databases work on the basis of managed serendipity, enabling multiple and unexpected encounters with original anecdotal material. As such they reflect natural learning processes, but with the advantage that we are not confined to people we can talk to as a source for stories, but have available all the stories ever told to the system. One growing area of application is for retired or retiring employees who will not write down what they know, but boy will they tell stories! Interestingly many people entering this area cannot resist the desire to interpret people’s stories. They want to tell employees which stories they should hear and what those stories mean. A true narrative database uses only original material and searches it based on abstract questions that discourage directed enquiries to create serendipitous encounter (Snowden 2001). For example “show me all the stories told by a naïve archetype around the theme of project failure told with emotional intensity from the perspective of a first witness with the intention of excusing failure”; a query that will then produce say 18 plus stories which are selected or the search criteria altered; maybe the naïve archetype is made more cynical and the third party perspective is sought. Narrative databases can be a first entry knowledge management system; observing the patterns of use can determine where investment in best practice might be best focused, with supporting anecdotal evidence in support.

Social Network Stimulation

Too many people focus on managing knowledge rather than managing the channels through which knowledge flows. Just connecting or linking people can be a major knowledge management activity. Mentors provide such functionality but new tools now allow us to telescope five to six years of social networking down to five or six weeks, albeit with less density. Such programmes aim to create linkages where no linkage currently exists and are particularly useful during re-organisations and activities such as merger and acquisition. The key point to emphasise here is that the learning model is top down in respect of the heuristics and boundaries that govern the creation of the social network, but the membership of the network is self generative and voluntary in nature. Attempts to engineer a network through design and allocation of staff to groups generally fail as they create artificial relationships that are not sustainable. Self selecting social network stimulation replicates, but in a shorted timescale a natural process. An observation at this point, a lot of KM practice observes natural phenomena and then tries to abstract them into a formal process. A lot of community of practices are set up on this basis. The problem with this is that the circumstances surrounding a particular natural process can never be fully known, we need to recreate the context to stimulate a similar occurrence, but as human systems are complex the stimulation will also produce a new pattern, hence the use of heuristics and boundaries to influence and direct the formation of those patterns.

Disruptive Pattern Breaking

A large amount of learning does not require us so much communicate knowledge be it best or good practice but rather to disrupt established knowledge. I have argued elsewhere (Snowden 2002) that formal communities of practice need regular and ritualised disruption to prevent entrained thinking and in consequence avoid the longitude problem. There is nothing as conservative as a deep expert! We can also introduce disruption in a narrative database by introducing unexpected material, say from history that creates a new perspective when a story about a current situation is encountered. In more advanced cases under development we are starting to build experimental narrative filters in which the user is forced to see things from radically different perspectives for application in everything from foreign policy to sales practice. Providing new perspective can create new understanding and prevent negative pattern entrainment. Other advanced applications utilise game environments working with science fiction writers and alternative histories to create a disruptive metaphor to allow people to encounter things indirectly through the metaphor rather than dealing with reality which can often be painful. This process of displacement leads to another narrative technique based on ancient practice in which archetypal story forms, utilising archetypal characters that have emerged from the water cooler stories of an organisation can enable people to confess to sin, without attribution of blame through the medium of stories told about the archetype.

Efficiency does not necessarily lead to Effectiveness

The main focus in process re-engineering and to a degree knowledge management practice has been in efficiency. The pursuit of efficiency lies at the heart of the concept of best practice, if there is a best way then it is surely more efficient for all agents within a system to follow it. Unfortunately while efficiency does achieve effectiveness in mechanical or highly structured human systems it does not in respect of the majority of human interaction which, as previously stated is complex in nature. An interesting feature of complex systems, particularly in social insects is that for a system to be effective there needs to be a degree of inefficiency in the operation of its agents. Humans are the same; the efficiency focus of best practice harms effectiveness because it assumes repeatable past patterns of cause and effect. Driving out inefficiencies increases vulnerability to new threat as the adaptive mechanism of the complex system has been withdrawn. Indeed I frequently argue that in using narrative we are building worst practice systems which are both more popular in facilitating voluntary access and more effective in creating learning within an organisation. Best practice has a space in knowledge management, but the space is small, highly specialised and generally expensive. Creating a learning ecology on the other hand that bounds but recognises diversity is another matter all together, here the dynamics of human interaction and enquiry can be built to permit both better decision making, and though the active management of serendipity the enablement of innovation.


Clark, A. (1997) Being There: Putting Brain, Body, and the World Together Again

Klein, G. (1998) Sources of power: how people make decisions” MIT

Snowden, D J (2001) Narrative patterns - The perils and possibilities of using story in organisations' in Knowledge Management Vol 4.10, Ark Group, July/August 2001

Snowden, D. (2002) “Complex Acts of Knowing: Paradox and Descriptive Self Awareness” in the Journal of Knowledge Management – Vol. 6, No. 2, (May) pp. 100-111

Weick, K.E. & Sutcliffe, K.M. Managing the Unexpected Jossey-Bass 2001

Dave Snowden is Director of IBM’s Cynefin Centre for Organisational Complexity. Details of the Centre’s work can be obtained from the web site or directly from the author via e-mail:

First published ARK Knowledge Management Page of

Details © D.J.Snowden 2003

The database is protected by copyright © 2017
send message

    Main page