Request a demo
Measuring learning effectiveness - improving the LMS

Measuring learning effectiveness - improving the Learning Management System

Traditional Learning Management Systems (LMS) have a limitation. They only record part of an individual’s learning, i.e. eLearning and maybe some classroom training.

However, new and more powerful, complete tracking solutions now allow a much fuller picture of an individual’s learning to be captured, including activities and learning done outside of the LMS. This means managers and individuals can make better, more informed decisions about learning and its effectiveness and measure it in far more sophisticated ways.

The Donald Kirkpatrick training evaluation model

This is probably the best known model for evaluating the effectiveness of training:


Whilst some businesses are good at measuring the bottom three levels of Kirkpatrick’s triangle, it’s often difficult for them to go beyond this. To be fair, even if training objectives have been clearly stated and designed, showing their return in terms of tangible, measurable business benefits (let alone return on investment (ROI), which is sometimes added as a fifth level to the triangle) can be difficult. It can even be difficult to prove that training has had a measurable effect on pre-defined key performance indicators (KPIs).

So, how can we better measure the effectiveness of our organisation’s learning activities? How can we spot our most successful courses and our best performing learners? And how do we factor in all that informal learning taking place away from the LMS?

There are two essential points to consider:

1. Knowing your users learning experiences.

As well as eLearning and mobile learning, recognise that a huge amount of learning is taking place away from the LMS: seminars, classrooms, social networks, books, conferences – to name but a few.

2. Know the characteristics of your top performers.

Why do they learn more effectively than others? Can you link their success to KPI’s or other measures? Are they better at their jobs? Answering these kind of questions would depend on what your organisation does. For example, in a sales-based business you may choose to look at metrics like conversion, conversion rate, sales and revenue made. In a service-based organisation you could examine customer satisfaction reports and call handling time. For project managers you could include information about projects finished on time and on budget, amount of profit made etc.

Gathering this kind of rich data means you can examine the learning activities of top performers, tie these together with their learning paths and experiences and compare them to others. Have certain KPIs improved after certain types of training? Are top performers taking more training courses and if so which ones? You now have the beginnings of a more tangible way of identifying and quantifying how effective particular training programs are – indeed, how effective your entire training program is.

The Experience API LRS, playing a central role

Recording all learning experiences, of all types, taking place within an organisation is where the Experience API (Tin Can) and most particularly the Learning Record Store (LRS) now enters our discussion. An LRS is, put simply, a central store of all learning records. A rich database that records learning experiences. Experience statements can originate from anywhere - not only your LMS. Example sources of learning statements could be:

1. Apps.

This could be an App that allows you to scan a QR code to confirm your attended a particular class or seminar. Or maybe that you read book, a blog or watched a YouTube video? Maybe this kind of App would prove that high performers make extensive use of external resources.

2. Browser widgets.

Like the App mentioned above, a widget at the bottom of your browser could signal you’ve read a blog, a Wikipedia article or watched a useful YouTube video. Even better, this information could be shared with other learners in your community.

3. Manual web forms.

Alternatively, imagine being able to click on a web form that lets you record and share your recent learning experience, details being entered by yourself or an instructor on your behalf.

4. Games.

It’s also technically possible to send learners statements to the LRS direct from games and simulations. This illustration shows the central and vital role that the LRS can play within an organisation:

The effects of correlating learning and performance

What insights can we gain from building up such an extensive store of learning activities on a LRS? Remember, having that data is only part of the answer. To use it effectively it’s important to define KPIs for individuals, teams and the business that we’re hoping to influence. In some cases, these identified KPIs and metrics can be also be stored on the LRS allowing even more useful data correlations.

Provided with all this useful rich, linked data, these are the sort of questions that can be answered:

  1. Do top performers use social learning for training more often than other forms of training?
  2. Which training paths have the most successful outcomes?
  3. What types of learning are most popular? Is there a correlation with individual performance?
  4. Are your most effective staff those trained via Webinars?
  5. Has the introduction of mobile learning improved sales team performance?
  6. What types of training paths have bad outcomes or don’t improve performance?
  7. What is the impact of a specific training program on the specific KPIs?

Joining these learning streams across an organisation would create an even richer set of potential insights.

Summary

Effective technologies (particularly the LRS) now provide organisations a route to measuring, learning and training effectiveness in a more concrete way than has been possible before now. Learning statements and data from entire organisations can be captured and stored centrally.

You can summarise the approach like this:

  • The LRS can be used to capture learning data from across all the learning activity that tales place within your organisation, not just within the LMS.
  • Define KPIs for the business, department and individuals.
  • Focus on the KPIs that the business wants you to influence.
  • Analyse the learning analytics to identify the patterns and the training paths that create the best, and the wort, outcomes.
  • Identify the learning habits of top performers.

Anyone involved in the world of learning and development would benefit from taking a close look at the potential offered by the Experience API and the LRS as it is significant. After all, the quicker you can start to gather rich, concrete information about your organisations learning activities, the sooner you can lick them into shape!

Alex Mackman
Author: Alex Mackman

This article was published by Alex Mackman on 23.01.2016. I'm Technical Director for Agylia. I'm responsible for the product strategy and operations side of our business. I'm passionate about technology - specifically how it can be applied to learning.

Some of our clients:
  • Microsoft
  • Atos
  • Deloitte
  • O2
  • E.ON
  • FCO
  • RBS
  • Virgin Trains
  • Morgan Sindall
  • QNB
  • NHS Digital