Ideas
Jed Emerson

UNREASONABLE.is ... for entrepreneurs hungry to put a positive dent on history ... The Metrics Myth—Why Quantitative Presentation of Qualitative Value Matters

Burgess COMMENTARY

Peter Burgess

UNREASONABLE.is ... for entrepreneurs hungry to put a positive dent on history Inside Our Programs Debunking Design For Impact Guest Post Metrics & Evaluation The Metrics Myth—Why Quantitative Presentation of Qualitative Value Matters 2 months ago41 Comments Photo from UnsplashPhoto from Unsplash

Since at least the early ‘90s, those involved in the generation of more than money have been on a quest to capture the right balance between documentation of impact and claims to same. For me personally, I have been a reluctant crusader in this quest for coming on thirty years. As a social worker in San Francisco’s Tenderloin District, operating a multi-service center for homeless youth, justification of our funding was often tied to service counts. And yet, while our program served large numbers of youth, we often found financial support for those youth tied to annualized government contracts and lump sum foundation grants directed to other programs for these funds did not follow performance, but were allocated on the basis of what I called a dance of deceit set to the music of politics, perception and persuasion.

Some years later, as founding director of the nation’s second venture philanthropy fund, I led the team that in 2000[1] introduced what I believe was the first formalized methodology to track SROI—Social Return on Investment. Prior to that, I remember hearing foundation and social investors alike state, “We invest for Social Returns!” Yet when queried as to how, exactly, they tracked such returns they would laugh, saying, “Oh, no! It’s a metaphor—we don’t actually track social returns!”

I never did understand this.

A traditional investor would never place funds in a venture that did not have capacity to track financial performance and, ultimately, financial return. How then was it okay to place funds in social enterprises (whether for-profit or non-profit) in the absence of reporting systems tracking the extra-financial value creation of that investment? Why bother calling ourselves social entrepreneurs and impact investors if we could not in some way document, differentiate and assess this added level of value creation?

Why bother calling ourselves social entrepreneurs and impact investors if we could not in some way document, differentiate and assess this added level of value creation? Tweet This Quote

Over past years, as I’ve both participated in and observed the evolution of metrics and impact reporting discussions and practice, I’ve come to realize the “metrics challenge” is something of a myth. It is a myth in both senses of the term in that on the one hand it is the Big Foot of impact investing—widely known yet seldom seen—and, on the other hand, pursuit of the next level of “metrics” is the Holy Grail of our professional quest, a story that has critical meaning, value and importance in our lives.

Big Foot Counts ... Couldn’t we create a metrics framework truly capable of capturing not only outputs in order to justify funding, but also assess the deeper outcomes of our efforts as well? Tweet This Quote

At its core, the metrics discussion is a myth because while everyone talks about it, professes to have seen it and debates its importance, impact metrics remain ephemeral, a force living in some deep wood, visions of which are obscured by branches, brush and bullshit. At a national metrics conference several years ago, I recall a breakout session presentation by one of the world’s best known impact investment organizations—one that appears on everyone’s list of favorite impact funds—listening to a nuanced and well-considered presentation by their head of impact performance. Following the formal PowerPoint show that included impressive definitions, charts and data, the presenter was asked, “How do these impact metrics inform your work?” to which the presenter responded, “Oh—we don’t actually use these metrics in our work. We just need them to give our funders!” After everyone had a good laugh, he said, “No, no—I’m serious—we don’t use them at all!” Peals of continued laughter echoed.

While I was taken aback by how forthright this practitioner was in his assessment of the relevance of metrics to practice (or lack thereof!), I also experienced something of a flashback to my time as a social worker wherein metrics mattered not as a means to more effective program or service delivery, but rather simply as a way to justify continued funding to those removed from the action. Why, I thought, would someone possibly spend their entire professional life involved in work the value of which could only be assessed on the basis of people served rather than lives changed? Couldn’t we create a metrics framework truly capable of capturing not only outputs in order to justify funding, but also assess the deeper outcomes of our efforts as well?

Too many social entrepreneurs and impact fund managers view metrics and performance tracking as “something we do for funders” Tweet This Quote

More recently, as various new impact funds have been conceived and launched, it has become clear, despite the old adage “You can’t manage what you don’t measure,” many impact fund managers do just that. While we all talk about the difference between traditional and impact investing as a function of our tracking the extra-financial performance of social and environmental value creation, too many social entrepreneurs and impact fund managers view metrics and performance tracking as “something we do for funders” rather than something we need do for ourselves to ensure our work is actually creating the impact we seek rather than the outcomes we claim.

And I believe it is getting worse.

In recent weeks, I’ve heard of two leading fund managers—overseeing hundreds of millions of dollars of supposed “impact” capital—who have said that since they are investing in sectors that have intrinsic impact, their funds (investing in sustainable agriculture or renewable energy or affordable housing or small business development or what have you…), by extension, do not need to document or be managed explicitly for impact. Now, perhaps I’ve been in these discussions for too many years and have grown both cynical and short-tempered, but how is this any different from nonprofit managers who are working to affect change in communities or in people’s lives asking us to trust them since, after all, “We’re all here for the children?”

We should not confuse intent with impact. Tweet This Quote

As our greatly revered 40th President of the United States used to say, “Trust—but verify!” and I think impact investors should do the same. Social entrepreneurs or impact fund managers who claim to have impact in the absence of documentation tied to their execution should be challenged. If they have the continued arrogance to state that their special fund has impact simply by virtue of their strategic intent, they should be tarred and feathered and shunned, for we should not confuse intent with impact. As one of my good colleagues once observed, “You could be creating the best and greatest ‘next generation’ battery for the electric car, but if your rare earth metals are being strip mined, you are not advancing a more sustainable planet.”

Period.

All too often, I fear, we do indeed confuse our noble intent with actual, realized and demonstrated impact. They are not the same.

Metrics matter.

All too often, I fear, we do indeed confuse our noble intent with actual, realized and demonstrated impact. Tweet This Quote

The challenge, of course, is that actually creating meaningful, relevant social management information systems to track the total performance of our impact investments takes more time and money than many social entrepreneurs and investors care to spend. In some perverse variation of the “We don’t fund overhead, only program” comment one sometimes hears from foundation executives and donors, too many of those in the impact investing community view an effective metrics reporting system as “nice to have” as opposed to “critical to our practice in advancing impact.” And therefore, while the topic of metrics continues to carry much weight in our conversations—a “must have” topic at any impact investing conference and justification for any number of stand alone meetings, events and professional junkets—the reality is for far too many of us involved in impact investing actually attaining sustained metrics practice is, in truth, frequently seen as the dreamlike, mythical Loch Ness Monster, with flashing skin and scales so close you can almost touch them as it twists and turns and slowly descends out of sight, into the muddy murk of our investing waters.

In Quest of the Holy Grail

And yet, the myth persists that we can attain our goal of effective and relevant metrics assessment and reporting. One must ask, after all the frustration and challenges, why do we bother? I submit we persist in our pursuit because we know at a deeply visceral level our goal of integrating meaningful metrics into the core of our efforts to create a changed world has value and is central to who we are.

When asked,

“Do you believe in a god?”

C.G. Jung replied,

“I do not believe there is a god—I know there is a God.”

And so here, my good and patient friend, is the fundamental point:

Our commitment to attaining the Holy Grail of credible, sound impact reporting is less a function of faith than of stripping our current intellectual and practical shrouds away to reveal the deeper essence of what we know to be true.

Simply because our present, dominant approaches to assessing metrics fall short of our task—How can one measure the full value of a life saved or possible future changed? What, ultimately, is the real impact and value created through the allocation of our capital?—we persist because we know two things:

How can one measure the full value of a life saved or possible future changed? Tweet This Quote

First, we know we are on a Hero’s Journey of inquiry and innovation. Too often we forget the present system of tracking financial performance (the basis upon which trillions of dollars flow through global capital markets and the foundation upon which too many of us build our lives) is the outcome of over sixty years of development, refinement and debate. In the U.S., GAAP and FASB (the fundamental building blocks of mainstream business and finance) were not created until after World War II; and it was not until the creation of the Environmental Protection Agency in 1970 that business and many nonprofits began tracking and assessing environmental metrics on a consistent basis. And while social metrics have always been a part of the parlance of government and philanthropic funding, many foundations and social investors have not sought to weave performance assessment into their process of allocating funds until recent decades.

We see the metrics mist clearing by the year. Tweet This Quote

It is for these reasons I am quite comfortable with the reality that those creating the metrics and evaluation frameworks of tomorrow will need another twenty years to build what is not yet ours, for I know it will come in good time.

Second, we are creating Total Portfolio Reporting frameworks to track the returns of unified investing strategies (capable of reflecting the aggregate performance of philanthropic, social and environmental value creation) because we know it can be done—and indeed, we see the metrics mist clearing by the year.

As initiatives such as

• The Principles for Responsible Investing’s Integrated Reporting work,

• the recently re-organized SROI Network,

• the Sustainable Accounting Standards Board,

• B-Lab’s B-Analytics framework,

• CapRock’s iPar system,

• the ANDE Metrics Working Group

and a variety of grassroots initiatives coming together around various sets of common reporting for assessing community impact, we find one can create a balance between our aspirations for a better world and the challenges of demarcating our progress toward that goal.

In the end, I hate the whole metrics debate.

One can create a balance between our aspirations for a better world and the challenges of demarcating our progress toward that goal. Tweet This Quote

It is repetitive, mind numbing and distracting from the critical task of fighting the forces presently destroying our societies and planet. Each time some ignorant (not stupid, mind you, and yet, not fully aware of what they do not know; they are quite rightly, ignorant) newcomer enters the discussion, we’re all expected to re-group and re-define concepts and issues well documented and explored in the past. The continual, mindless reminders that not everything that counts can be counted leave me frustrated and even angry at some who for reasons beyond me don’t seem to understand that such now trite insights were the very starting place of this journey well more than 25 years ago and that, indeed, as newcomers they are as far behind the current exploration as we are from our goal.

Yet, we make progress despite our doubts and complications.

We advance the practice of both impact investing and performance measurement one step forward and two steps back as the current “knowledge” of the crowd Tweet This Quote

We advance the practice of both impact investing and performance measurement one step forward and two steps back as the current “knowledge” of the crowd actually pulls us backward to previous thinking and practice. And we know the appropriate application of metrics bring meaning and insight just as they demonstrate the limitations of such efforts. This journey is critical to us not simply because it holds the promise we may, in time, create effective tools to better represent the quantitative performance of qualitative value, but rather because in more deeply being a part of this journey toward documenting the impact of our strategies, we see new and more nuanced aspects of the value we seek to create through the course of our lives…

We learn more about who we are as individuals and as a community.

And so I say, Onward, compatriots, onward!

There is another windmill over yonder hill—

and we will make our own path by walking toward it!

[1] Our original framework was presented in 1996 in the Roberts Foundation report, New Social Entrepreneurs: The Success, Challenge and Lessons of Nonprofit Enterprise Creation, but we did not present the full complement of a framework and related tools until our Box Set publication in 2000.

This originally appeared on Jed’s blog Facebook146Twitter