Evaluating eLearning in a Crunch

By Shelley A. Gable

For anyone who’s in the midst of designing an eLearning course, have you figured out how you’re going to evaluate its success?

- Some of you are proudly nodding.
- Some may be thinking that it’s not necessary.
- Others would like to evaluate, but lack the time and/or know-how to do so.

For those of you in the first category...kudos. You’re onto something. I hope you’ll comment on this post with some of your reflections and advice on evaluation.

For those in the second category...consider that evaluation allows us to confirm with data that what we’re doing works. Without some form of evaluation, how can we know for sure? Besides, presenting data-based evaluation results to stakeholders helps build our credibility.

Now about that third category...let’s spend some time on this. Though much of what’s written about evaluation tends to make it sound like a huge undertaking, it doesn’t have to be. A small-scale, simple evaluation can be better than no evaluation effort at all.

If you think you might be able to spare a few hours on what could prove to be an immensely valuable activity, read on.

First, identify the questions your evaluation should answer.

Take a few minutes to brainstorm. Maybe even ask a colleague or two to brainstorm with you. Even if you identify several questions you would like to answer, you can trim your list down to a few priority questions to keep the evaluation effort small and manageable.

Consider questions like:
- How easy was the eLearning course and its activities to use?
- Which objectives did learners struggle to accomplish?
- Did learners perform as expected on the job?

Next, determine who can help you answer the questions and how you can get that data.

Back to brainstorming.

If your time is limited, think simplicity. For instance, no rule says that a questionnaire must be long. If you opt to survey learners, consider putting together a questionnaire with a handful of straight-forward Likert-scale questions (perhaps with one open comments area at the end). Ratings on even a few items should give you a sense of how favorably learners tended to view the course.

Did your eLearning course include quizzes or knowledge checks that you can pull results for?

Need to get a sense for on-the-job performance? While there are many elaborate ways to do this, you have simple options too. If you’re dealing with performance measures that are already tracked, it may be as simple as requesting the appropriate reports and asking someone to spend an hour teaching you how to interpret them. If the performance measures aren’t so clear, you might send a quick email to learners’ managers asking for their impressions.

Now, collect the data and analyze it.

If you’re in a crunch to do this quickly, don’t worry about dusting off your statistics textbook. Keep thinking simple. For instance, you might tally the number of favorable and unfavorable responses related to your original questions. Or simply categorize the comments you received to help you identify trends.

Finally, share your impressions.

An important component of any evaluation effort is communicating the results. Admittedly, if the evaluation was quite limited in scope and followed some of the loosely structured advice above, you probably can’t claim to be able to tell the entire story. But that’s okay.

As part of your communication, openly discuss your limitations. Tell your stakeholders what questions are still unanswered and any new questions that have been raised. You might even recommend additional evaluation measures as next steps.

If stakeholders get a taste of what an evaluation can reveal and they want to know more, your small-scale efforts might earn you resources for a more thorough evaluation project.

What else?

What evaluation options do you use that are simple, quick, and informative?