Ideas
??? (from LinkedIn

Impact Investment & Grantmaking: Measuring Impacts & Outcomes ... some ideas about measurement in education

Burgess COMMENTARY

Peter Burgess

Impact Investment & Grantmaking: Measuring Impacts & Outcomes

Mission-driven investment, whether it's a grant from a non-profit or funding from an investor who is interested in 'doing well by doing good,' want to actually measure what their donations and investments have produced. This is both to have a sense of how much has been accomplished and also to keep track of what is effective. It's important learning for the community.

Funders (Impact Investors) want outcomes (impacts) to be measured, but those who receive the funds lament the workload involved in meeting accountability demands.

If you think specifically about what is being measured as being a category of intangible asset, then you can get a better sense of what it is that you need to measure. Intangible assets exist in the intellectual, social, and physical domains. You can learn them as knowledge, as a process, or as a system.

By creating a grid that examines these domains (intellectual, social, and physical) with respect to the three scales of individual, knowledge, and systemic we can come up with outcomes that are extremely helpful for the mission of the organization.

Let's do an example in the area of education.

What is the mission of education: to educate. What does that mean?

That means some combination of:

• Intellectual knowledge (traditional pedagogy); process (how to study); and systems (how to think)
• Social knowledge (how to behave); process (how to solve problems); and systems (how to build effective groups)
• Physical knowledge (food pyramid/health); process (how to play, what are the rules); and systems (how to train)

IMAGE

How we weight each of those things is and always will be under dispute. In fact, whether they're taught at all in the schools is under dispute. The only thing that's agreed upon (and even here the specifics are disputed) is that kids should learn some facts while in school.

Nevertheless, of those nine parameters, the first column is often agreed upon -- in generalities -- by the population at large: you go to school to learn the curriculum. Yet if you talk to the most educationally sophisticated parents, they often evalute educational environments based on the third column -- and you can't get to the third column without also learning the second one. I've color coded Green, Yellow, and Red to indicate how thoroughly we're measuring those metrics.

Cool Story Bro, but What Does that Have to Do with Outcomes?

Let's take those nine parameters, and think about what metrics we might want to use to see if the mission is being met.

Intellectual Assets

Traditional Curriculum - Formal testing.

How to Learn - Testing the process. Currently, I have heard of classes and tutors who offer process coaching, but regular classroom teachers also do binder checks, require turning in rough drafts, and other process training. Students' study process (or lack of one) accounts for a lot of performance disparity in older students.

Metrics need to be thoughtful and differentiated: Everyone gets a daily binder check until they manage 20 perfect ones in a row, and then weekly, and then monthly. How compliant is the teacher in doing that process? If the students aren't progressing, it's important to look at why -- and the teacher nagging is usually not going to solve this issue, as organization skills are complex.

How to Think - Critical Reasoning is tough to teach and tough to measure. You toss a problem to students that should be above their knowledge level, and see how they approach handling their lack of knowledge.

Science sometimes teaches this explicitly; math certainly does a little in the context of how to construct proofs. Debate (Forensics) is similar. Some excellent English teachers do; and a few kids actually take philosophy or other social sciences. But other than the context of the curriculum, it's not often measured explicitly, even in science.

In order to measure critical reasoning explicitly, we need to develop complex rubrics that illustrate progression along a trajectory, like we did with the simpler case of binder checks.

Social Assets

How to Behave - One of the reasons Early Childhood Education works far better for lower socioeconomic families is because it teaches 'school readiness,' which are behaviors that aren't as common in families where parents are working as laborers in the service industry, where different norms than 'sit still and listen' are important to success. The best predictor for a child's academic achievement is the academic achievement of the primary caregiver. How to behave at least well enough for teachers to teach is a cornerstone of schools.

Unfortunately, beyond basic self-control, we minimally teach -- let alone measures -- positive behavior.

There are programs such as Mindful Schools that focus on personal calming. Teaching project management could address positive behavior beyond the simple kindergarten rules and into the more complex junior high and high school social milieus: how to include, to facilitate?

How to Solve Social Problems - In some excellent schools, this is taught within the context of the classroom as issues arise, and/or it is taught in the abstract while discussing literature in high school English or past conflict in History courses. In many others, well, we all know about playground justice and how it can become fatal. Explicit conflict management instruction, and later project management instruction could help children learn skills at an emotionally neutral time -- not while in conflict. The skills require training and intervention, though, and without staff, this is a very difficult thing to do.

Further, just like the example about how critical reasoning is taught, some community education is necessary. If parents model ineffective or socially destructive methods of problem solving, they could both benefit from the instruction, and the children need to be guided in how to reconcile what they observe and what they are taught. Still this is a sensitive subject, as an article about revenge by Jared Diamond so eloquently described. The primary bargain is that justice will be provided by society, one way or the other.

(I have long felt this area is the justification for bringing games into the school environment. Kids are highly motivated to keep playing, so when conflict arises they have profound incentive discover a solution. Being able to describe the conflicts and how they were resolved during a post-mortem would be an enlightening -- and measureable -- exercise.)

How to Form Groups - Collaboration is, like conflict reconciliation, a daunting educational goal. Because of that, it also needs to be addressed explicitly to some degree.

Conveniently, this can also be done within the context of project management by learning how to form groups and assign tasks according to skills and abilities. We admire students who have the innate ability (or parents who teach them) to form teams and clubs. There is no reason why this can't be taught, and the effectiveness measured.

Physical Assets

Last, and easiest, measuring physical improvements has somehow gone out of fashion in many places, but I wish it wouldn't have. I believe socially we have gotten past the value-judgments of the 1970's and can address reasonable improvement as the goal, rather than explicit performance.

In fact, this is the easiest realm to measure. Yet we forsake it because somehow it seems more obvious to those in charge of such things that physical prowess is a talent that some lack; of course, so is intellectual prowess.

Final Thoughts about Education

I have been very fortunate that learning has never been difficult for me. But I have known many people for whom school has been a daily pounding of the 'reality-check' that they just weren't particularly smart. Not smart, not popular, not sporty... just plain average in a highly competitive world.

It's important to have metrics in place not only so that we can create an educational system that is itself a non-judgmental, data-driven learning environment, but because it's a healthy part of human nature not to remain in an environment where we feel alienated and beaten.

We need to measure and identify for all students what it is that they contribute positively, how they're going to make their way in society, and where they can fit in on a team.