ROI is a False Number: Moving towards Evidence-based Learning

Doug Lynch (The University of Pennsylvania – The Wharton School of Business) was one of the keynote speakers at Sunday night’s kickoff to the Learning 2007 conference, and he was quite obviously the smartest man in the room — so naturally, I flocked to his session this morning.

> “Thomas Jefferson once observed the increasing strength of evidence for Isaac Newton’s theory of gravitation that Newton ‘indulged in reason and experimentation, and error fled before them.'”

Newton developed theories, and some of them have held up. LET professionals have also held up theories, and some of them work, if you don’t sweat that some parts hold up better than others. But people talking about principles as “facts” aren’t facts — they’re just assertions.

There are no scholarly writings on ROI on Google Research. None. People have been able to demonstrate the economic returns on learning, but there is more than what can be demonstrated in these studies. Zagat ratings are coming out now for health care, so there’s evidence to support the “effectiveness” of the quality of health care, but it may not be all the evidence you want.

So here are some questions to help deconstruct ROI:

* What level of sophistication do you need in terms of understanding evidence of impact?
* What tools do you use to gather evidence of impact?
* What tools do you use to analyze performance / learning?
* Do you use any methods to evaluate implementation in addition to the intervention? If so, how?

What forms of evidence do you need to determine if what we do impacts business goals?

ROI has a very narrow meaning to CLOs, Finance people and Economists. As a learning profession, we throw around the term ROI, when we should really be talking about “Impact.” The person hearing you when you use the term “ROI” is thinking something completely different.

A problem with our world:

> Man is naturally metaphysical and arrogant and is thus capable of believing that the ideal creations of his mind, which express his feelings, are identical with reality. From this, it follows that the experimental method is not really natural to him (Claud Bernard, 1865).
> If you want to find out what is going on, you need to look at what is going on (Yogi Berra)

Learning is loosely coupled. What LET professionals *do* is a messy business. We have norms in our world

Kirkpatrick’s Four Levels

* Reaction of student
* Learning
* Behavior
* Results

So… what are some underlying assumptions about ROI?

Ratio of Net Benefits: ( Benefits – Costs ) / Costs

Let’s deconstruct benefits (or profit). What goes into calculating the benefits of training?

* Sales?
* Income?
* Stock Price?

The idea behind ROI is that it is a number that is monetized. You define it, you measure the costs and the benefits and then you monetize it. Are these things easy to do? No, they’re not. Stock price is a net-present value of what the company is doing today and it’s speculation on how it will do in the future. Stock price also gets at some other things that are going on — like a company’s performance in relation to the rest of the market.

For you to say that you have impact, you need to have control of everything that’s going on. To use ROI as a measure of learning’s impact, you’d have to control the entire market and factor non-related issues out — everything from competitive advantages, the cost of raw materials, fuel costs… you get the idea. And even if you could factor those things out… you still have to factor out all the human-factors to filter down to just Learning’s impact on an organization.

The relationship of Learning departments and HR departments probably is a better indicator of Learning’s Impact on an organization. Generally, the quality of your students is an indicator of their future performance.

A few definitions:


1. Ground for belief; testimony or facts lending to prove or disprove any conclusion
2. Information, whether in the form of personal testimony, the language of documents, or the production of material objects, that is given in a legal investigation, to establish the fact or point in question


1. Practice founded upon experiment and observation

It might be interesting to look at both when looking for the impact of learning on an organization. Gather the evidence. Be skeptical. If the evidence negates your hypothesis (or the goal of the learning project), go down the trail of questions as to why? Research it. Embrace the situation and question if the training was actually bad, or if maybe it’s an implementation issue. We are all academics as LET professionals. We research. We question. We need to apply it when it comes to organizational impact.

**General Principles for Research**

1. Pose significant questions that can be investigated *empirically*
2. Link research to relevant theory
3. use methods that permit direct investigation of the question
4. Provide a coherent and explicit chain of reasoning
5. Replicate and generalize across studies
6. Disclose research to encourage professional scrutiny and debate

**Stop thinking of Learning as a solo-artist**. *Learning in an organization is part of the symphony* that lends to performance.

There’s no magic number, calculation, bullet to prove your impact on performance. Build the case with a preponderance of evidence to get to **Impact**.

Impact: there’s no “definition” of impact, because it’s contextual to the organization. But one way to think of it is a preponderance of evidence and empirical study that make the case for what you’re doing in relation to the organization’s business goals. It’s not something that can be monetized, but it is something that an organization can “define” for itself. Reach for the available metrics — sales numbers are there, and you can measure them pre and post. How can you measure Leadership? You can’t.

One reply on “ROI is a False Number: Moving towards Evidence-based Learning”