Learning Measurement: Top Tips From Industry Experts

Last summer, I started a new series that highlights the best advice I’ve heard from guests on our podcast, The Talented Learning Show and our long-running, live monthly webinar series. I planned to share tidbits in 5 posts focused on the following learning systems use-case categories:

• Extended Enterprise Learning
• Partner Training
• Non-Profit and Association Member Education
• Customer Education
• Commercial Training and Certification

But while reviewing podcast transcripts, I noticed a pattern I had previously glossed over. Regardless of the core topic or use case, every conversation eventually leads to learning measurement.

The need for accountability has always been the lifeblood of extended enterprise education. But now, employee and academic learning professionals are also just as interested in results-driven instructional strategies and programs. And the vendor community is responding in spades.

As an independent instructional technology analyst and consultant, I’m seeing a surge in solutions and services designed to make outcomes easier and more effective to measure and understand.

It’s exciting to watch learning measurement come full-circle, as it pulls training into the world of modern data analytics. And along the way, more training professionals are finding a seat at the executive table. That’s because they’re equipped to demonstrate impact using the logic and language that business decision-makers appreciate.

Of course, there are countless ways to skin the learning measurement cat. But below are bits of wisdom worth taking with you, no matter what tools or techniques you use when tracking, evaluating and improving results.

“Best Advice” for High-Impact Learning Measurement

From Episode 8:
What Are the Business Benefits of xAPI?

Q: We hear a lot about xAPI. For those who want to use it as the basis for a  data-driven training and development culture, how do you recommend they get started?

A: Mike Rustici, Founder and CEO of Watershed LRS

Most importantly, don’t be intimidated. Don’t be afraid to start small and dive-in.

Using data isn’t necessarily about demonstrating a big seven-figure impact on the bottom line. Even the smallest finding can have a big impact. It may be about identifying no-show users at last Thursday’s training session and helping them find another path to learning that expands their professional knowledge and skills. Ultimately, this may help improve their lives. That’s real impact.

Essentially, the learning measurement process is about relying on data to hold yourself accountable and strive for continuous improvement. So find a place where you can get started. You don’t need a new full-on learning ecosystem to start using data and gain insights from what’s available.

There are many resources, especially about xAPI and related learning measurement tools. Our team is always happy to help any of your listeners get started, if they want to know a little more.

From Episode 13:
How Can a Compliance LMS Reduce Business Risk?

Q: You see many big global companies that have sorted out their compliance training infrastructure. But what would you tell those who feel stuck with their current infrastructure and want a new approach?

A: Bill Mastin, former SVP, NetDimensions (now PeopleFluent)

I would say three things:

1) Make sure your learning measurement solution simplifies reporting, so full visibility is available within only a keystroke.

2) Put a thought process in place that focuses on measuring training all the way to business impact. For instance, we integrate with the WatershedLRS via xAPI, which lets you correlate what happens in an LMS with business results. But no matter how you do it (LRS, xAPI or some other way), you need that capability.

3) You also need clear alignment – from the CEO and all executives and down to everyone across the company – so they understand the risk and reward of being an organization that values compliance.

If your organization focuses on these three things, you will do very well.

From Episode 17:
How Do New Analytics Tools Elevate Learning Measurement?

Q: Advanced analytics tools are on the cutting edge, but what do you see ahead?

A: Tamer Ali, Co-Founder and Director of Authentic Learning Labs

We will be able to answer questions more quickly, John. Questions that business people ask. For instance:

• Which training catalog issues need immediate attention?
• Where do weak enrollment, declining engagement and poor scores indicate a need for intervention?
• What similarities exist among those who aren’t finishing courses?
• When people have expressed negative sentiment, is there a better time to promote specific content to them?
• Who are the window shoppers? What common attributes can help us prompt them to commit and engage?
• What topics are outdated? Should we mothball content that hasn’t been touched within a specific timeframe?

Others look at the score on the performance side. Who are the outliers? Where are they performing well or poorly? What topic areas would they be good at? What topics would be of greatest interest to them? This kind of predictive analysis helps organizations improve how they develop and market courses.

Actually, we’re answering these questions right now. But in the future, you can expect to see data-based insights helping organizations understand these factors faster and make better business decisions more quickly.

From Episode 19:
Why Is Customer Education Gaining Momentum?

Q: Customer education professionals seem obsessed with measurement. What metrics do you use? And what would you tell others to measure, so they can prove value?

A: Dave Derington, Senior Manager of Customer Education at Outreach.io, a sales engagement platform.

Metrics. This is a sticky challenge. All across an organization, people want to know, “How is training actually helping our business?” So it’s important to think about ROI.

You can focus on calculating the cost of delivering training and then determining how to cover those costs. But I tend to focus on consumption and adoption. Are customers using the content? Are they struggling with anything? Do I have an equivalent of Net Promoter Score (NPS)?

For example, let’s say you complete my product 101 course, John. Ideally, I’d include an NPS question at the end, asking you to rate the value of that course on a scale of 0 to 10. How useful was the content to you? Why or why not? I’d ask for specific feedback.

For additional data points, define who’s consuming training. What patterns do you see? How many people participated in training? How are they using your product before and after consuming your content? What are they struggling with? In a B2B setting, do training participants tend to be associated with specific clients? Do you see increased product usage among organizations that rely on customer training, compares with those who do not?

Once you tie together participant feedback with broader contextual data, you can start developing conclusions about business impact. And then you can start sharing your message.

You probably won’t be able to guarantee that customer training will drive 100% increase in early-stage product use. But you should be able to talk about the kind of lift that training brings to product adoption and ultimately, customer success.

From Episode 32:
How Do You Build a Business Case for Customer Training?

Q: For those who are building their first customer education business case, what do you recommend to make this process less intimidating?

A: Bill Cushard – General Manager of Learndot by ServiceRocket

Start with a hypothesis and a goal. You need to get that right. Your hypothesis should be some version of, “If we do customer training, we know something is going to get better.” You need to define what that ‘something’ is.

Then set a goal by quantifying how much that ‘something’ should be. For example, you might say, “This year, we think we could train 10 more new customers than last year, to increase their product usage rate to at least the same level as this year’s trained customers.”

Next, test your hypothesis by starting small. Create a model where, without asking for upfront money, you can develop a proof-of-concept. Open a Zoom account or use another simple video conferencing tool and offer live online training. You’re probably comfortable with live training, so start there. Don’t start with videos. Plus you’ll get immediate feedback from your customers and reps.

You could probably rough-out a sample class right now. Maybe you can only deliver one hour-long session a month. Fine. Do that. Or you might want to call a customer and invite three people from their team to join you for a live online training session. Treat it like a focus group.

Whatever you do, ask for feedback. And teach that class again and again. Keep iterating and improving based on what you learn. But here’s the point: You can’t move quickly if you spend three months just developing and producing a single video.

So do this instead: Develop a hypothesis. Set a meaningful goal. Then start simple and cheap.

From Episode 33:
What’s the Best Way to Measure Training ROI?

Q: What steps would you recommend to training professionals who want to build their financial and business acumen?

A: Ajay Pangarkar, President of Central Knowledge

L&D professionals don’t need to be financial experts. But they should be financially literate. If you’re trying to promote learning to others, then you should be a learner.

Learning is not just about learning in your own space. It’s about reaching outside of your sphere and thinking more broadly about your profession.

If you’re working in a corporation, it’s your responsibility to understand the business – but not to be a business expert. There are people who are hired to do that, just as you are hired to be a learning expert. But you need to understand the business and how you fit in that space.

So take some easy steps. Nothing complicated. First, look online for articles from sources like TalentedLearning. John, you write great blog posts with valuable insights. There are a lot of other free resources. My favorite is eLearningIndustry.com.

There are many great books available, also. I would start with books recommended by learning industry associations. For example, I’ve written The Trainer’s Balanced Scorecard, which speaks to the relationship that connects learning with organizational strategy.

Actually, 30 years ago, Kaplan and Norton sparked the balanced scorecard performance framework revolution. And another author, Paul Niven, wrote a step-by-step book about this performance management approach, which translates Kaplan and Norton’s work into an easy read.

I can rattle off a lot of stuff, but people just need to get out of their silos and expand their horizons. Pick credible sources. There are a lot of writers and speakers who pretend to know what they’re talking about.

But you can’t go wrong by relying on Harvard Business Review or Stanford Business. They offer free articles that will open your eyes to business functions and strategy, so you can make your learning programs more impactful and more precise.

I guarantee that if you do this, you’ll be successful. You may only improve your professional competence by only 5%. But even a fractional improvement like that can make a huge impact.