Two steps forward, one step back

 Josh and Footprints by Vu Bui

Josh and Footprints by Vu Bui

I have a sign that hangs by my desk. It is simply black print on a white piece of paper. It says: The Prime Directive: Improve outcome performance measurably. That’s it. Nothing more. Nothing less. In my opinion, it sums up my job quite nicely. The problem is that I can’t tell when I am successful and when I am not successful. Have you ever had that happen to you? Have you ever worked for a company and been unable to tell if you were successful or not in executing the mission statement?

My goal with each and every elearning course or training tool that I develop is make a positive impact on the user and to improve their performance. Sounds easy. Unfortunately, there are so many factors involved that it is difficult to pinpoint changes in performance that are directly related to the course. I have tried pretests and posttests. I have tried focus groups. I have tried coaching sessions, all in an effort to zero in on the true impact of the learning interaction. So far, I have not found a satisfactory was to track the impact of the training interventions that I have designed.

To date, the most rewarding feedback is simply annecdotal – the people in my Leadership Development Program (LDP) say that the courses and discussions are helping them with their daily management tasks. They have changed the way that do certain things based on the material that they learned in the program. Maybe this is enough. Maybe I am just being too picky in looking for a “scientific” way to measure this impact. Obviously this measure of performance change is all the more important in today’s economy in which the learning and development folks must show results.

How about the rest of you? Have you experienced this challenge as well? How did you measure performance change? Do you have any ideas to share that might help other L&D folks out? Let’s hear them everyone!

Advertisements

4 responses to “Two steps forward, one step back

  1. Zed, I think your dilemma is a common one, and I don’t know that there’s any easy solution. The farther up Kirkpatrick’s levels you go, the more it costs and the less direct the connection to a specific learning event.

    If your LDP grads are back on the job, and still saying the course made a difference in their daily tasks, that can’t be bad. I’ve done what I called “Level 1.5” surveys — sent a list of specific outcomes to grads and asked them to rate their current ability on a 4-point scale (can’t do – can do easily; one guy added a 5: “can do in my sleep).

    That’s anecdotal, but it’s post-training anecdotal. In this case, the skills were specific (“sync your client data [with the online system],” for example).

    The client organization had other, more indirect data about the overall adoption of the system that the training dealt with — so they could see patterns of usage, increase in market share that they attributed to adoption of the system.

    You might informally talk to some managers of your grads, asking about their criteria related to exemplary performance.

  2. Dave,

    Great tips and advice as usual. I appreciate the ideas you put forward. I was able to have the LDP participants take a management skills inventory before the program and I plan on having them take it again after the program. I also had their managers fill out the same assessment and they will fill out a post assessment as well. My plan is to do this on a 30, 60, 90 day basis. This might give us some insights as to the impact of program. Again – at least in anecdotal form. We’ll see what happens.

    Thanks again for the input.

    Zed

  3. Ah, you seek the Holy Grail of measurement, watch out for the killer bunnies while on your quest.

    I think your anecdotal feedback is what you base your effectiveness on. The softer the skills, the more difficult it is to determine how much of an impact training had. There are too many outside factors that can influence soft-skills training.

    For things like software training, it’s pretty simple – can the person perform the task after training? Leadership is a totally different ballgame.

    For leadership training I’d talk to the people under your students; the ones being managed. They will tell you what kind of leader the person is. Ideally, talk to them before training and then several months after.

  4. Gary (or shall we call you…..TIM!),

    First of all – nice Monty Python reference! Right on target. I appreciate your thoughts on this. I had not thought about going to the team members of the managers (D’oh!) It might be a little bit more tricky politically, but I think it is important.

    Thanks again for sharing.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s