Replying to LO25281 --
As a metaphor.....and related to training's impact... when you turn a
smile upside down you get a frown...and vice versa.
Hypothesis: there is a direct and positive relationship between evaluation
cost and the level of evaluation examined.
This hypothesis should be difficult to disprove. Regardless of the
evaluation model used.
As we examine ROI, we must calculate evaluation as part of the
intervention cost and affix it to the evaluation plan (part of the ISD or
design, development, implementation model you use...). That is, the
evaluation (ROI) of an intervention should be viewed in a systemic and
holistic manner. The plan should associate the essential or core
competencies (knowledge, skills and attitudes) and prioritize efforts. In
some cases it might be wholly proper and appropriate to measure at level I
while at another time at level II and so forth. We spend so much of our
time at level I because it is less costly and more appropriate (a greater
amount in the former versus the later). Up front association of evaluation
cost and level to competency need is required. Even in programs where the
evaluation model is well developed and integrated into the design plan,
the evaluation effort (the ROI determination) is reduced or dropped right
out when it comes time to expend the resources. One reason for the ease of
reduction or cost might be that we argue so strongly for higher levels of
evaluation when it they are not needed.
Here might be an example. Cultural awareness in Saudi Arabia. Western
society (U.S., Canada, Europe, etc) people employed in SA must have an
orientation on the cultural aspects of living there. Do's and don'ts. When
one thinks about the importance of this bit of training, it seems
appropriate to evaluate the effectiveness of training at level III or IV.
But like growing more mature and learning public manners, experience and
observation are the most effective teachers. In the "manners" case, you
are taught not to eat with your elbows on the table and you see that
others do not. In SA you are told, as a male, not to cross your legs and
show the sole of your shoe or foot. One might say, ah ha, the observation
is level III and IV. Indeed it is, but that observation didn't cost much
at all. And it cost less if it were not done in a contrived setting, for
the purpose of evaluation -- it was not a formal evaluation. In the class
we ask if the training was enjoyable and if the learner understood the
information (levels I and II). We spend time and money on these two
levels. To many would argue that the unscientific observation at the table
or in the living room does not help evaluate the training. Perhaps not of
an individual person, but it should of a population.
The bottom line seems to be that we don't well articulate the needed or
desired level of evaluation, set against the core competencies and gaps.
We try to measure to much, to rigidly.....with to much rigor.
>and the things I remember learning most from in my life which made the
>biggest difference to who I am and how I deliver I would probably have
>said at the time I did not enjoy .......... so the smile sheet would have
>written them off....
>and smiles for one person are not for another..........
"Swan, Steve R. SETA CONTR" <SwanSR@ftknox5-emh3.army.mil>
Learning-org -- Hosted by Rick Karash <Richard@Karash.com> Public Dialog on Learning Organizations -- <http://www.learning-org.com>
"Learning-org" and the format of our message identifiers (LO1234, etc.) are trademarks of Richard Karash.