Developmental evaluation, a form of real-time evaluation focused on innovation and complexity, is gaining interest and attention with funders, program developers, and social innovators. Yet, it's popularity is revealing fundamental misunderstandings and misuse of the term that, if left unquestioned, may threaten the advancement of this important approach as a tool to support innovation and resilience.
Developmental evaluation (DE) is an exciting advancement in evaluative and program design thinking because it links those two activities together and creates an ongoing conversation about innovation in real time to facilitate strategic learning about what programs do and how they can evolve wisely. Because it is rooted in both traditional program evaluation theory and methods as well as complexity science it takes a realist approach to evaluation making it fit with the thorny, complex, real-world situations that many programs find themselves inhabiting.
I am seeing and hearing this term throughout the non-profit, government and civic engagement worlds and I ought to be excited at seeing DE brought up so often, yet I am often not. Why? Is it that I have some Charlie Brown syndrome, or is it something else?
Alas, with rare exception, when I hear someone speak about the developmental evaluation they are involved in I too often fail to hear any of the indicator terms one would expect to accompany stories about such an evaluation. These include terms like:
Alas, DE is starting to follow a well-worn path laid by terms like systems thinking, which is getting less useful every day as it starts being referred as any mode of thought that focuses on the bigger context of a program (the system (?) -- whatever that is, it never seems to be elaborated on) even if there is no structure, discipline, method or focus to that thinking that one would expect from true systems thinking. In other words, its thinking about a system without the effort of real systems thinking. Still, people see themselves as systems thinkers as a result.
The term DE is following this pattern and is being used more frequently in a manner that some might call cavalier, but that I suspect reflects aspiration rather than reality.
That aspiration is about wanting to be seen (by themselves and others) as innovative, as adaptive, and participative and as being a true learning organization. I don't think anyone is lying, rather I think they've been caught up in using terms that haven't been given enough thought.
DE has the potential to support all of the abovementioned things (innovation, learning, participation, adaptation and more), but to accomplish these things requires an enormous amount of commitment. It is not for the faint of heart, the rigid and inflexible, the traditionalists, or those who have little tolerance for risk.
Rolling up the sleeves to do DE
Doing DE requires that you set up a system for collecting, sharing, sensemaking, and designing-with data. It means being willing to -- and competent enough to know how to -- adapt your evaluation design and your programs themselves in measured, appropriate ways. This is tough stuff and well worth the effort, but tough nonetheless.
DE is about discipline, not precision. Too often, I see quests to get a beautiful, elegant design to fit the 'social messes' that represent the programs under evaluation only to do what Russell Ackoff calls "the wrong things, righter" because they apply a standard, rigid method to a slippery, complex problem or organizations think its something they can tack on to the end of a project.
It's so much more than that, which is maybe why I get a bit concerned with the language.
Why does this fuss about the way people use the term DE matter? Is this not some academic rant based on a sense of 'preciousness' of a term? Who cares what we call it?
It matters because the programs that use and can benefit from DE matter. If its just about gathering some loose data, slapping it together and saying its an evaluation and knowing that nothing will ever be done with it (or maybe some dumb things will be done with it), then maybe its OK (actually, that's not OK either -- but let's pretend here for the sake of the point). When real program decisions are made, jobs are kept or lost, communities are strengthened or weakened, and the energy and creative talents of those involved is put to the test because of evaluation and its products, the details matter a great deal.
If DE promises a means to critically, mindfully and thoroughly support learning and innovation than it needs to keep that promise. But that promise can only be kept if what we call DE is not something else.
That 'something else' might be a form of utilization-focused evaluation, or maybe participatory evaluation or it might simply be a traditional evaluation model dressed up with words like 'complexity' and 'innovation' that have no real meaning in the way they are used.
(When was the last time you heard someone openly question what someone meant by terms like innovation or complexity when they used it? It's because its used so loosely that we forget they are actually quite sophisticated concepts in practice.
We take such terms as given and for granted and make enormous assumptions about what they mean that are not always supported).
There is nothing wrong with any of these methods if they are appropriate, but too often I see mis-matches between the problem and the evaluative thinking and practice tools used to address them. DE is new, sexy and a sure sign of innovation to some, which is why it is often picked as the term/method/tool to use.
Yet, it's like saying "I need a 3-D printer" when you're looking to fix a pipe on your sink instead of a wrench, because that's the latest tool innovation and wrenches are "last year's" tool. It makes no sense. Yet, it's done all the time.
There is something alluring about the mysterious. Innovation, design and systems thinking all have elements of mystery to them, which allows for obfuscation, confusion and well-intentioned errors in judgement depending on who and what is being discussed in relation to those terms.
We are now seeing developmental evaluators who have almost no concept of complexity, service design, and have completed just a single course in program evaluation. I'm seeing traditional organizations recruit and hire for developmental evaluation without making any adjustments to their expectations, modes of operating, or timelines from the status quo and still expecting results that could only come from DE. It's as I've written before and that Winston Churchill once said:
I am always ready to learn, but I don't always like being taught
Many programs are not even primed to learn, let alone being taught. Why? Because that is hard stuff and something we need to invest more than words in.
What to offer and what to ask for
So what should someone look for in DE and those who practice it? What are some questions those seeking DE support ask of themselves?
Of evaluators
Of programs
Of both evaluators and program stakeholders
DE is not a magic bullet, but it can be a very powerful ally to programs who are operating in domains of high complexity and require innovation to adapt, thrive and build resilience. It is an important job and a very formidable challenge with great potential benefits to those willing to dive into it competently. It is for these reasons that it is worth doing and doing well.
In order for us to get there this means taking DE seriously and the demands it puts on us, the requirements for all involved, and the need to be clear in our language lest we let the not-good-enough be the enemy of the great.