What's in my head today?

This month’s Learning Circuits Big Question is, “How do you assess whether your informal learning, social learning, continuous learning, performance support initiatives have the desired impact or achieve the desired results?”

I apologise for the delay in offering my two cents’ worth. Putting the finishing touches on our new electronic performance appraisal system and preparing our National Learning and Development annual conference has taken up most of my time over the past few weeks, but it’s time to knuckle down now and make my monthly contribution.

So many words have been written about assessing learning’s impact and whether it has met it’s goals that it’s almost impossible to contribute something new to the field. So, rather than offer some kind of academic paper on the hows and whys of learning assessment, I’m just going to give you some insight into how we do this at TSAEP, and my thoughts on measurement of success in general.

What are we hoping to achieve?

Of course, to be able to measure any kind of impact or effect, we need to decide up front what vale our intervention (whether it be a formal course, a social learning initiative or a project aimed to embed new opportunities for collaboration) will add to the business. Where will we see improvement as a result of this initiative? After identifying the effects we expect to see, we need to be honest with ourselves and our stakeholders, and keep to that yardstick. We can’t change the goals as the project rolls out if we see we aren’t having the impact we’d planned. There may well be benefits we didn’t anticipate or plan for at the beginning, and we should make note of these things in any follow-up report on the effects of the initiative, but we shouldn’t shy away from facing up to the fact that an initiative didn’t have the desired effect. That’s critical information that should drive a look into why this may have occurred. It’s worth noting here that a project may well be deemed a success from an organisational perspective, even if we didn’t have the impact we’d planned, provided the positive unanticipated effects are relevant enough to the business.

The problems with measurement

I feel compelled to add a rider at this point. The information above makes it seem that learning initiatives have easily-defined start and end points. This is not the case, and it is especially not the case with informal and collaborative learning initiatives. We can take some measures of impact at various points in time, but these programs will continue to evolve long after the measuring is done. It’s critical that we not only remember this, but help the business stakeholders understand it as well.

One of the key criticisms of the Kirkpatrick model of learning ROI is that it treats learning as something that starts and ends at obvious points. The model was developed at a time when learning simply meant formal courses that were deemed to be over when the participants went back to work and started to apply the knowledge or skills learned. This assumption was never really true in any case, but it’s position as the basis of the Kirkpatrick model makes that model less and less relevant in today’s environment, where collaborative and social learning is finally being recognised as “real” and valid initiatives in the corporate world.

So how can we measure?

Having trashed the model that remains the most widely used tool for the assessment of learning programs, what are we left with? The fact that learning is an open-ended, lifelong activity doesn’t mean that nothing can be done to gain information on whether what we are doing is having a positive effect. We can still take some point-in-time measurements, provided we bear in mind this is just what they are – a snapshot of a certain situation at a given point in time.

1. Organisational performance

From the organisational perspective, there is really only one measure that counts – is whatever has been done having a positive impact on business performance? This is the gold-standard that CEOs, Directors and other managers look for in any initiative, whether it’s learning, organisational change or anything else that’s added to the working environment.

When I’ve talked about this very point to training managers over the past few years, a disappointingly high proportion of them either argue against it or simply switch off to a message they don’t want to hear. Common arguments against it include, “We can’t be held responsible for business performance because we don’t control what people do in their day-to-day roles”, or “We don’t control the tools and the data to tell us whether core business performance indicators are changing in either direction.” Jay Cross, in his book Informal Learning, talks about speaking at a conference of training managers about this very point, and watching one attendee walk out after muttering something along the lines of “I didn’t get into training to worry about business results.” The belief that there is some kind of separation between the activities of a training department and improvement in business performance is a dangerous one; those who hold it shouldn’t complain when their department is the first to suffer budget cuts when their business undergoes inevitable belt tightening.

The business unit responsible for learning needs to arrange access to business performance information and also to learn what measures are important and why. This may involve having direct access to performance data, simply having access to those that can run these reports, or something in between (access to certain information and the ability to have someone else source other data when needed), but however it’s done the members of the training department need to demonstrate an understanding of the core business data that defines the organisation’s performance, and they need to have the ability to relate what they do to that data in order to provide evidence of the impact of learning initiatives on the numbers that matter to managers.

2. Quality measures

The business data and the information that flows from it is critical to the training department’s ability to make the case for the value they add on the ground. There are also other measures that may be more or less relevant to executives and other managers at various times. This includes information of retention, staff satisfaction and compliance with internal or external requirements (e.g. anti-discrimination training, occupational health and safety). If a CEO or board is concerned about an organisation’s high turnover rate, evidence of an improvement in that figure following an intervention such as bullying awareness training (if that is an issue) can support the learning and development team’s argument that they have made a positive difference to a clear business need. It doesn’t take a genius to realise that happier workers who feel adequately supported are also more productive than those labouring in a negative environment.

There are also some measures that are meaningful to the training team, though they may not be of interest to others. Here, we’re talking about data such as participant satisfaction with the services offered by the team, utilisation of available resources and feedback regarding the currency and relevance of information released by the team. These are really customer service measures that can help the training department identify whether there are development needs within the team itself. While the executive team are unlikely to be interested in receiving this information, it is important to the manager of the learning and development department.

Measuring what matters

At the end of the day, it’s important to be able to define the information that will be most relevant to the leaders of the business, how the training team can have an impact on that information and to ensure the team has the ability to report on it as needed. Some learning and development managers fall into the trap of measuring those things that are easy to measure (e.g. figures such as number of hours of training offered, the number of staff trained) and provide these to executives in reports as if they mean something to the bottom-line performance of the business. Concentrate on the things that business leaders concentrate on, and develop your ability to elucidate a clear link between what the training team has done and how the numbers move up or down. This is the only way learning leaders are going to earn a seat at the executive table when important decisions are being made, and at the end of the day, improved business performance is the reason learning and development teams exist at all.Other measures (e.g. quality measures, above) are important to the training department, but are often much less important to the executive team; and while it’s tempting to report on those things that are easiest to measure, doing so only promotes the idea that the training team doesn’t understand the business.