The Value of Estimation in Learning Impact Measurement

Jan 11, 2017

dont underestimate the value of estimation in learning impact measurement

Advances in learning technology, brain science, and teaching methods over the past few years have created new possibilities for developing knowledge and skills in the workplace and not a moment too soon. A majority of respondents in a recent McKinsey study expect significant changes in corporate learning over the next three years. The demand for online learning at the speed of business is growing in the quest for organizational agility. [1]

Learning and development achievements, learning benchmark reports In many of these enterprises, the gap may be more due to lack of measurement that poor execution. What gets measured gets noticed, and few L&D organizations measure the impact of learning on the business. It's hard for CLOs to measure what they don’t manage, and most of the data they need live outside of their control. Because senior leaders and line-of-business managers do not see information about the impact of learning, they may have only a vague notion that the connection exists. For most organizations, the road ahead is challenging. According to McKinsey, less than half of L&D organizations align themselves with corporate priorities to help their organizations meet strategic objectives. Towards Maturity’s 2016 benchmark survey of over 5,000 organizations reported shortcomings in L&D’s achievement of five essential outcomes, and contrasted the average assessment with that of Top Deck (top 10%) organizations.

Partnerships in Measurement

If L&D wants to show the impact on the business, they will have to make the first move toward a partnership in planning, executing, and measuring learning outcomes, and that means devising measurement models where metrics don’t yet exist.

Much of today’s learning happens outside the LMS. Most organizations do not yet have sophisticated methods for capturing offline and informal learning. Control groups are almost always impractical, and historical data for trend line analysis rarely exists. Fortunately, expert estimation is easy and inexpensive to implement.

Expert Estimation

Using the expert estimation approach to measurement does not mean you gather people together and have them guess at results. Estimation is a learned skill. In learning programs, the experts can be the participants. They may be the most knowledgeable and will have a vested interest in the impact on their performance. Their estimate will likely be credible in the eyes of management.[2]

To be successful, you will need to lay the groundwork for the measurement effort.

  1. Engage the executive or manager who owns the process. L&D will act as a consultant, and the responsible executive or manager, acting as a sponsor, will own the execution and measurement. Work with the sponsor’s team to develop and deliver the learning initiative.
  2. Identify the direct links between the learning interventions and the business outcome. Determine how you will measure and report results.
  3. Inform participants they may be asked to take part in a focus group or questionnaire to evaluate the impact of learning on their job performance. Help them identify all the factors that may have an impact. This will prepare them to isolate the factors to improve the value of their feedback.
  4. Implement the plan and measure the results. (we are assuming an improvement.)
  5. If possible, use small focus groups to gather data. If focus groups are not workable, you can use a questionnaire.

Estimations are a percentage of improvement and the degree of confidence in the estimate. Let’s take an example of a learning initiative to improve customer service with blended learning and performance support. Suppose a participant estimates that 30% of the improvement in performance is due to a single factor and expresses a 90% confidence in the estimate, the adjusted percentage of increase in that factor is 30% X 90% = 27%. If the monetary value of the gain is $100,000, the value of that component of the learning intervention is $27,000.

Measuring the success of learning through technology

Developing participants into better estimators does not require a substantial investment in training. Udemy has a two-hour course online on “How to Estimate Anything” for $10, and there are hundreds of other online courses available.


An Estimate Example

Every business runs on estimates. Even when we make decisions on “gut instinct,” we are calculating risks and probabilities, sometimes unconsciously. Helping people to become better estimators can often be a matter of explaining the principles.

Here’s an example. Hold one arm straight out in front of your body with your thumb pointing straight up. Now, how far is your thumb from your nose? How confident are you that it is less than six feet and more than six inches? You would probably say you are 100% confident. Now, how confident are you it is 28 inches and less than 32 inches?

By starting with a large range of values and gradually reducing it, you can reach a point where you have substantial confidence in a narrow range of values. When you express an estimate in terms of confidence in a range of values, people immediately grasp the value of the estimate and are willing to act on the estimate.

Open the Door to A Learning Organization

Other measurement methods may be more accurate, but expert estimation, used wisely, can get the job done faster and at much less cost. It may be the key that opens the door to building a learning organization.

References :

1. Benson-Armer, Richard, Arne Gast, and Nick Van Dam. "Learning at the speed of business." McKinsey Quarterly. May 2016. .

2. Elkeles, Tamar, Patricia Pulliam. Phillips, and Jack J. Phillips. Measuring the success of learning through technology: a step-by-step guide for measuring impact and ROI on e-learning, blended learning, and mobile learning. Alexandria, VA: ASTD Press, 2014.

Pixentia is a full-service technology company dedicated to helping clients solve business problems, improve the capability of their people, and achieve better results.

Learning and development with Sumtotal learning

Previously:  Next up: 


News Letter Sign up

Get in touch with us
phone_footer.png  +1 903-306-2430,
              +1 855-978-6816