William M.K. Trochim
Concept Systems Incorporated

THE PROBLEM: WHAT EVALUATION NEEDS TO ADDRESS

Like most things, evaluation can be viewed very differently by different audiences. Some think evaluation only involves looking at the outcomes, results, or effects of some program. But this is a rather narrow definition that can lead to evaluations that neglect crucial parts of the overall picture. In a broader view, evaluation involves the assessment of the entire educational program process from the initial conceptualization and development of the program to any demonstrable long-term effects. It is this broader vision of evaluation that corporate universities need to embrace. In doing so, we need to resist the temptation to narrow the scope of evaluation, thereby limiting its potential effectiveness. With that in mind, it is important to consider the full range of issues evaluations should address.

THE ROLE OF EVALUATION IN CONCEPTUALIZING THE TRAINING PROGRAM. Too often, evaluation enters the picture "after the fact" to assess eduacational programs that have already been through initial development and testing. In reengineering corporate university evaluation, we need to move evaluation forward in this process, including evaluation activities even as early as the initial conceptualization and development of the educational program itself. At the initial stages, various stakeholders have different views, often unarticulated, about what the training should consist of, how much relative emphasis should be placed on different topics, and what kinds of outcomes are desired. Partners, program developers, and in some cases, even clients will hold views that sometimes contradict or conflict with each other. Evaluation is an essential tool for describing the views of these various stakeholder groups and helping develop a consensus that can form the basic conceptual framework for the program. At each stage of the conceptualization and development process, we can use simple, efficient, structured evaluation activities to gather information and feed it back to the various stakeholders in order to facilitate a common understanding of the training program.

The first essential conceptualizing task an evaluation must address is the description of the key components of the training program. What are the central topics the training should address? And, are there differences among key stakeholders about what the central topics should be? Second, evaluation should help determine the relative importance of each training component or topic. In any training endeavor, there will always be some topics that are more important than others, that will be given greater emphasis than others. The delineation of relative importance is an essential part of determining the unique ‘fingerprint’ that the program will have. Here, as elsewhere, it is essential for evaluation to describe whether there are consistent differences among stakeholders in the relative importance attributed to different topics. Third, key stakeholders should describe the outcomes they expect the training program to have. Evaluation can be used to help identify potential short, intermediate and long-term outcomes of interest and the different ways these might be operationalized for measurement. Finally, it is likely that different stakeholders will have different ideas about how the training program will affect the outcomes. Given the training components and their relative emphasis, and the potential outcomes, evaluation can help stakeholders articulate their expectations for the training, or describe the relative effect the program will have across different outcomes. There should be a documented link between a training component and an outcome. If we have an outcome is not able to be linked to a training component, why would we think that it would be affected by the training at all? Evaluation can help articulate these conceptual linkages.

THE ROLE OF EVALUATION IN MONITORING THE IMPLEMENTION OF THE TRAINING PROGRAM. Once conceptualized, a program typically goes through a considerable period of development and revision. Evaluation plays a crucial role in this ‘shakedown.’ But it is important to notice that evaluation’s job is much easier at this stage if the conceptualization phase was done well. Assuming that evaluation was utilized in the conceptualization phase, we would enter the implementation phase with some valuable information: a detailed delineation of the topics that should be covered in the training; an estimate of how much relative emphasis each topic should be given; a list of the potential outcomes of the training; and, an estimate of the relative effectiveness of the training on each outcome.During the implementation phase, evaluation can be used to begin to measure each of these four things. On the program side, simple benchmarks can be constructed to show that the intended topics were covered, and with the right amount of emphasis. On the outcome side, evaluators can begin to develop and field-test measures that reflect the full range of expected outcomes and assess relative impacts. When a rich, detailed conceptual basis is available, the feedback that evaluation can provide during the implementation phase will be particularly valuable for diagnosing difficulties in the training and fine-tuning it in subsequent manifestations.

THE ROLE OF EVALUATION IN ASSESSING THE EFFECTS OF THE TRAINING PROGRAM. Once the program has been field tested and readied for wider application, evaluation has a critical role to play in assessing how well the training works. In order for this assessment to be both efficient and effective, it is critically important that it be built upon the prior conceptualization and implementation efforts. Measures developed and tested during the initial implementation of the training can now be applied to assess ongoing preformance. Some of these will assess the ongoing implementation of the training, providing some quality assurance that the training continues to be implemented as intended. Some will examine the range of potential outcomes to see if they are affected in the manner originally envisioned. The results of these analyses will be presentable in terms that will now be familiar to all stakeholders because they were directly involved in the original articulation of the conceptual framework and the implementation. Additional measures will most likely also be designed at this stage to look at the longer-term impacts of the training, or at impacts based on more objective measures than originally obtained (e.g., client assessment, return on investment).

THE SOLUTION: CONCEPT MAPPING AND PATTERN MATCHING

Evaluation needs to provide a wide variety of methods and supports in order to support so broad a range of tasks. Historically, each task and subtask of the evaluation endeavor was addressed with a different tailored approach. Program conceptualization and development used a different set of methods from implementation assessment which, in turn differed in approach from outcome assessment. This resulted in inefficiencies, lack of standardization, and the need to reinvent the evaluation process each time it was applied. There was no overarching strategy that pulled all of the evaluation pieces together and enabled greater standardization and re-usability.

Concept mapping and pattern matching are general approaches to evaluation that address effectively the need to evaluate the whole program from inception to effect. They add value to the evaluation process by providing an overarching framework that helps standardize the evaluation process and assures portability of methodology from one context to another. The concept mapping and pattern matching approach does not detract from the strengths of prior evaluation capabilities. Rather, this approach helps makes the disparate pieces common in previous evaluation work come together in a conceptually sensible whole. Concept mapping and pattern matching enable us to articulate, understand and communicate the value of training.

This concept mapping and pattern matching approach offers a number of benefits. It is integrated into the entire lifecycle of the training program, from initial conceptualization and development through assessment of short and long-term outcomes. It is general and flexible enough to apply to the evaluation of virtually any training program. The same sequence of procedures can be used in virtually any context, with slight adaptations possible if desired. It enables a rigorous assessment of both expectations and outcomes. Most evaluation work concentrates on the outcome side. But the concept mapping and pattern matching approach builds into the process consideration of what various stakeholders think should and will happen as a result of training. Consequently, the evaluation of outcomes is continually referring back to what was originally intended. Finally, the concept mapping and pattern matching approach adds value into the evaluation process without subtracting anything. It is fully compatable with other traditional evaluation approaches, but goes beyond them by organizing them into an overarching seamless evaluation strategy.

In the next section, the concept mapping and pattern matching approach to evaluation in a training environment is described in detail.

Methodology

THE CONCEPT MAPPING AND PATTERN MATCHING APPROACH TO EVALUATION

Concept mapping and pattern matching are two distinct methods that are most effective when used in concert with one another. Concept mapping is a process that enables a group of stakeholders to lay out their ideas on any topic. The result of the process is a map that shows the group’s ideas, how they are interrelated, how they are organized into more general concepts, and the relative importance of the ideas in the eyes of the group. For training evaluation, concept mapping will be used to map the training program and the outcomes expected from training. These maps accompany the training program from initial conception through final impact assessment, providing the detailed description of what the training consists of and the range of possible outcomes. The maps will help to guide what is implemented and measured, and they provide a visual device for graphing the ongoing progress of the training.

Pattern matching is the method used to assess the training program at all the different phases of its development and dissemination. Pattern matching always involves a comparison of patterns. Sometimes these patterns will be the weights different stakeholder groups think various training topics should receive. The match would involve looking at the similarities and differences between the groups in what they think should be emphasized in the training. Sometimes the patterns will consist of the outcomes different stakeholders think will or should occur as a result of training. In other cases, the patterns will be measures or estimates of the implementation of the training or of the outcomes that might be affected. The match might involve comparing what stakeholders thought would happen with what actually results. Pattern matching allows us to look at the overall training program at the same time that it provides very specific feedback about what subareas or suboutcomes are being implemented or affected more thatn others.

Because the methods are distinguishable from each other, concept mapping and pattern matching will first be described separately, with some preliminary examples obtained in training pilot studies. Then, the steps in the overall concept mapping and pattern matching method as it might be applied in a corporate university will be described.

WHAT IS CONCEPT MAPPING?

In many ways, a concept map is like a geographic one. At a glance, you can see where everything is. You can immediately tell how close any two locations are. You can quickly visualize a variety of paths that could be taken from one location to another. And, you would be foolish to set off on a trip without one. The major difference between the two types of maps is that the geographic map has locations on it while the concept map has ideas.

At first, the notion of an idea map or concept map might seem strange. We have gotten along fine until now without needing such a map. But consider how strange geographic maps must have seemed when they first became commonplace. Imagine some peasant farmer in twelveth century feudal Europe. For the most part he tended his land and lived his life within twenty or thirty miles of where he was born. How might he have reacted to the idea of a geographic map? "What would I need that for?" he might ask. "I’m not going anywhere anyway." The need for a geographic map was created by the increase in mobility in society. In a similar manner, we have not needed a conceptual map prior to the age of the information superhighway. The maps that each of us carry in our heads (whether actually drawn or not, these cognitive structures are there internally) have been sufficient for us to function adequately. But in the information age, where knowledge and communication merge, we need ways for different stakeholders to negotiate the conceptual territories they individually perceive. Concept mapping is the leading method for depicting the conceptual structures that groups and individuals have.

Why are concept maps useful? Certainly the old saw about a picture being ‘worth a thousand words’ is relevant. A concept map contains an enormous amount of information. It shows all the specific ideas (up to 100) that pertain to the topic of interest (e.g., the training program). It shows which ideas are more related to which others. It shows groups or clusters of ideas -- the more general concepts into which they organize. It shows how much importance or emphasis each idea and cluster of ideas has. A map is a much more efficient way than a set of tables or listings to organize and present this amount of information. And because it is visual, stakeholders are able to remember more of the information than if it were presented in the linear tables comon to older paradigms.

Another way to think about the usefulness of concept maps is to consider of the usefulness of their geographical counterparts. Imagine that you have to go someplace you’ve never been and that you stop to ask a stranger for directions. Almost invariably, their directions will be in a ‘linear’ form -- a sequence of steps that you must follow to reach your destination. You might be told to go down to the next major crossroad and make a left turn, go two blocks to the corner with the red house on it and make a right turn, and so on.

An error on any single step will throw off all subsequent steps and will almost certainly get you lost. Now cosider that you have a geographic map to use. The stranger giving directions can show you where you currently are and where your destination is, and can point out several alternative routes indicating which would be best under the circumstances. The map is more valuable because it provides both the overall context as well as the specific alternative paths. A concept map has similar advantages over more traditional methods of generating and organizing ideas. These tradiditonal approaches -- brainstorming, brainwriting, nominal groups, focus groups, Delphi techniques, and others -- result in lists or tables of ideas organized in one specific way. They are like the linear directions a stranger might give. The concept map, in contrast, shows all of the ideas at once, locating them in an overall context and suggesting alternative ways of depicting or organizing the specific ideas.

A CONCEPT MAP EXAMPLE. To illustrate what a concept map is, an example is provided that was generated in a brief pilot study. The purpose of this pilot was to develop a map that showed the major outcomes that are expected from a corporate university's education programs generally. The major stakeholder participants were the corporate university board members (approximately twenty) who took part in the semi-annual board meeting. This pilot was completed during three one-hour sessions on three consecutive days. In the first session, the partners brainstormed 80 different potential outcomes such as ‘skilled people,’ ‘one global culture,’ and ‘people who know how to learn.’ In the second session, each participant grouped the 80 outcome statements into groups of similar ones and rated each outcome for how important it is relative to the rest. In the third and final session, the concept map results were presented to them and they discussed their implications.

The accompanying figure shows the final concept map. The 80 separate outcome ideas are not shown on this map because the level of detail would be too great. What is shown are the ten clusters of the 80 ideas. These include concepts like ‘skills,’ ‘global culture,’ and ‘ROI.’ To see what’s ‘inside’ one of the clusters, consider some of the 80 statements that are in the cluster Skills:

	skilled people
	plentiful skills
	knowledge skills
	teaming skills
	professional, managerial, and leadership skills
	personal development
	people who know how to do the job
	before they start doing it (job ready)
	lateral thinking skills

The ten clusters of ideas were further grouped into four major ‘regions’ labeled Equity, People/Skills, Infrastructure and Return. Finally, the number of layers in each cluster shows the relative importance the participants attributed to those ideas. Here, Learning Strategies and Recognizing the Skills That Clients Need were given relatively high importance ratings while Organizational and Technological Capability and Risk Reduction were considered less important outcome areas.

A map like this can be used in a number of different ways. The stakeholders themselves might be surprised to see how their ideas are represented. As in a Delphi process, this feedback might prove valuable in shaping subsequent group opinion. Since this represents the views of only one stakeholder group, it might be useful to compare these results to others to see if their is agreement on the major outcome concepts and their relative importance. The map provides an organizing structure for thinking about the corporate university's outcomes. We could construct a simple MIS based on this structure and use it to depict the results of outcome assessment. Each of the original 80 outcome statements could be operationalized as a specific measurement or indicator. The results of such measures could be aggregated into the ten cluster areas and depicted as an outcome map which would have more cluster layers for outcoe clusters that showed greater effects of training.

The map shown here is an ‘outcome’ map, and a general one at that, because it considered the outcomes that would be expected across all of the corporate university's training rather than outcomes specific to one training program alone. In evaluating one training program, we would involve a number of different stakeholder groups (e.g., partners, program developers, clients) in generating both a program map that describes the specific activities and components of the training and an outcome map that depicts the outcomes specific to that training effort.

Although this was just a simple pilot exercise that took less than three hours of participant time, it should be apparent that with a relatively small initial investment a concept map can be developed that can be used as a conceptual organizing device and presentation format for all subsequent evaluation efforts in that area.

WHAT IS PATTERN MATCHING?

Pattern matching is an entirely separate method from concept mapping, but is most effectively used in conjunction with it. Pattern matching always involves a comparison (the match) between two or more arrangements of objects (the patterns). We can use pattern matching to compare the expectations of two distinct stakeholder groups, or to compare expected results with actual outcomes. Pattern matching gives us an overall estimate of the relationhip between the patterns and also shows us which parts of the patterns link up well and which do not.

Pattern matching has its roots in the problem of trying to tell the ‘forest from the trees.’ In a typical evaluation we would measure a wide range of potential training outcomes. When we look at the resulting estimates of effect, we can consider each outcome individually or we can list them in a table of results. But looking at each of these individual outcomes (the ‘trees’) would not give us any idea of the pattern of effects. Why is that important? Because training effects don’t occur in a vacuum. Outcomes are not independent of each other. When we look at the whole pattern of effects rather thatn at each one individually we are viewing the effects within a context rather than as separate and unrelated consequences of training.

Another way to think about patterns and pattern matching is to consider how we perceive a continuous picture when our eye is presented with a sequence of dots, as in a computer bit map or a dot matrix printer graph. If we look only at one dot at a time we can never understand what the whole picture is about. The set of dots as a pattern tells us something entirely different from each individual dot. We might still learn something useful by looking at individual dots, but we will never see the picture if we don’t look at the dots as a collection in their broader context.

In pattern matching we are performing an activity that our minds do very naturally -- we are comparing our mental map of the world with a map of the external world itself. Imagine using a geographic map to travel to a destination. You reach a crossroad that appears confusing and so you stop to check the landmarks to make sure that the direction you want to go on the map corresponds with the direction you are about to take in reality. You are matching two patterns -- the pattern on your map and the evidence in the real world. In training evaluation, there is a direct analogy. One pattern is the expected outcomes or effects of training -- the mental map that corresponds to the geographic one. The other pattern is the actual estimates of the effects or outcomes that are obtained through measurement -- the pattern in reality. Pattern matching in this case involves comparing the theoretical expectations with the observed outcomes.

We never expect that every possible outcome will be affected the same amount by our training. Some variables should be strongly affected, others hardly at all. The pattern or distribution of effects across outcomes is like a unique signature or ‘fingerprint’ of the training. When we look at the whole pattern of effects, we are looking at the entire fingerprint rather than just one small line of it -- we are looking at the forest rather than at the trees. And, just as with a bitmap image, if we only look at the individual outcomes, we will never be able to see the overall picture. Pattern matching enhances our ability to assess the effects of training by enabling us to see something entirely different, the overall picture, or the forest. It does this without jeopardizing our ability to look at individual outcomes as we’ve traiditionally done.

A PATTERN MATCHING EXAMPLE.

A small pilot study can be used to illustrate the idea of pattern matching. The study was designed to map the outcomes of a newly developed Object Modeling School. Program Developers made up the stakeholder group used to develop the map of outcomes. They generated 60 outcomes which were mapped into seven outcome clusters. The cluster layers on the top map show how much importance this developer group felt each topic was given in the training. After an initial beta testing of the training program, five participants were asked to rate where they felt they stood before and after training on each of the 60 outcomes. Gain scores were computed from these ratings and the average gain for the seven clusters is shown in bottom map. If the training works as intended, we might expect that we would observe the greatest gains in those areas that were most emphasized in the training. A visual comparison of the two maps shows that the patterns match -- in general, clusters which were more important in the training also showed greater gains. The match suggests that the training is effective in achieving its aims.

In addition to this overall pattern match, we can look more specifically at the results for each cluster or outcome area. By lining up the training importance ratings on the left and the observed outcomes on the right, we can see that the overall correlation between the two is quite strong (the correlation, r, is .89). Furthermore, we can see that in the outcome area Understanding the Approach, there was a stronger gain than we might expect from the importance that was associated with it in the training. Similarly, although Analysis Object Modeling was the most important area in the training, it did not turn out to be the area with the greatest gain. The overall strong pattern match suggests that the training has an effect. The specific matches for outcome clusters suggests areas where the program might be refined or modified to improve subsequent matches. Pattern matching provides both the general picture (the forest) and programmatically relevant targetted feedback (the trees).

Of course, these results must be viewed sceptically because they were based on a pilot study and extremely small beta test sample of five trainees. But they illustrate how the pattern matching approach assesses both the overall outcome of a training program while simultaneously providing outcome-specific feedback useful for revising and enhancing training efforts.

HOW THE CONCEPT MAPPING AND PATTERN MATCHING MODEL WOULD WORK AT A CORPORATE UNIVERSITY

HOW CONCEPT MAPPING WORKS.

Several key issues need to be decided before a concept mapping process can begin. The focus for the mapping needs to be developed. In a training context, there are typically two basic types of maps. Either the map will be focused on the elements or components of the training program itself, or it will focus on the potential outcomes of training. Usually we will use a heterogeneous group of stakeholders as the participants in the concept mapping process. Depending on the specific training program and and the availability of the participants, the process can be accomplished solely at face-to-face group meetings, entirely through remote communications via computer network and/or mail, or through some combination of direct meeting and remote communication.

Step 1: Generate the Ideas. The basic concept mapping process can be accomplished in three major steps. First, the participants generate a large set of up to 100 statements that address the focus. If this is a training program map, each statement would describe a specific component or element of the training. If it is an outcome map, each statement would describe a specific potential outcome. The statements can be generated in any number of ways. The simplest is through a face-to-face group brainstroming process. Alternatively, the statement set could be the result of a computerized brainstorming session (e.g., using Lotus NOTES or Ventana’s GroupWare), extracted from a development team committee meeting, derived from a focus group or nominal group process, and so on. The only criterion is that the final set of statements all address the focus for the concept mapping.

Step 2: Organize the Ideas. In the second step, each participants helps to organize the statements by performing two simple tasks. First, the statements are printed onto separate cards and each person sorts the statements into piles of similar ones, supplying a short name or label for each pile. Note that this is a similarity sort, not a priority one -- statements should be grouped together in the same pile if they are similar in meaning to each other. Each person can have as many or few piles as desired.

Second, each participant rates each statement on one or more scales of interest (see How Pattern Matching Works below). For instance, in a training program map each participant might rate each statement on a 1-to-5 scale of emphasis where a 1 means the topic should be given relatively little emphasis in the final training package and a 5 means it should be given extremely high emphasis. In an outcome map, a 1 might be used to indicate an outcome that was not likely to be affected much by the training while a 5 would indicate an outcome that would have an extremely strong training effect.

Step 3: Understand and Use the Map. In the final step, the participants interpret the map that is computed from their data. Usually a facilitator guides the group through an interpretation of the various forms the map can take, helping assure that the participants understand where different types of ideas are located and what major concepts emerge on the map. In some cases, the analyst will interpret the map for a more formal presentation to participants who will then concentrate on the use of the map in subsequent efforts.

HOW PATTERN MATCHING WORKS.

The pattern matching process is even simpler to understand than concept mapping. Essentially, two steps are required: the generation of the patterns, and the match itself. A pattern is simply the measurement of some variable of interest across all items on a concept map. Usually one of the patterns will be a theoretical variable, such as what a group of stakeholders believe or expect will or should happen. For instance, we might measure how much emphasis a specific stakeholder group thinks should be placed on each of the training topics. Often, we would collect this theoretical pattern through the rating task of a concept mapping process as described above. The other pattern can be either theoretical or observed. If both patterns are theoretical, the pattern match is addressing the degree of consensus across different stakeholder groups. For instance, if we had partners rate how much emphasis they think should be given to each topic and have program developers rate how much emphasis they believe they gave to each topic in designing the training, the pattern match would tell us about the degree of agreement or consensus between these two groups of stakeholders. We can call this a consensus pattern match.

When one of the patterns is theoretical and one is observed, we are usually performing either an implementation or outcome pattern match. For instance, if we had partners rate how much emphasis they think should be given to each training topic and we actually measure how much emphasis is given in the implementation of the training (e.g., through counts of the number of times the topic is discussed or the number of minutes it is being addressed), we are performing an implementation pattern match that tells us how well the partners’ expectations about what should be emphasized are echoed in reality. On the other hand, if one pattern is a program developer rating of how much they think each outcome will be affected by the training, and the other pattern is an estimate of the gain on each outcome six months after training, we are conducting an outcome pattern match that tells us about the effects of the training.

Although the idea of pattern matching is simple to comprehend, obtaining the patterns themselves can constitute a complex and challenging measurement task. Clearly, the validity of any conclusions we might reach will depend on the quality of the measurements we use. An outcome pattern match that uses objective measures of on-the-job performance will undoubtedly provide stronger evidence for effectiveness than one that relies on trainee self-assessments. Nevertheless, more rigorous measurement is usually considerably more expensive and time-consuming. Deciding on the level of measurement used in any specific evaluation will require a delicate balancing of rigor and cost.

We can think of the possible measures that might be used as differing along several important dimensions. First, measures can differ in their degree of objectivity. Self-ratings and anecdotal evidence falls on the subjective end of the continuum. Observed frequencies of some behavior or tallying of total income from an ingagement are more objective. In general, more objective measures are considered more rigorous and valid but are usually more expensive to achieve. Second, measures can differ in the point-of-view from which they are taken. We can think of this as a dimension that runs from those most directly involved in a program to those least involved (but affected in some way). In a training context, the trainees are at one end of the continuum, representing those most immediately influenced by the training. On the other extreme might be the clients in an engagement, or even the client’s clients. Third, measures can differ in terms of their time frame. On one end of this continuum are measures taken immediately after (or even during) the training itself. On the other would be long-term measures of outomes months or even years after training.

In any specific training evaluation, it will be desirable to collect a range of measures, with different types of measures collected over the lifetime of a training program. During the early stages of development, pilot testing and debugging, we would emphasize the collection of relatively inexpensive measures that are more subjective, from the trainee’s point-of-view, and collected immediately after completion of the training (the ‘a’ cube in the diagram). For instance, we might have each participant rate how competent they believe they were both before and immediately after training on each of the specific training outcomes. As the training program stabilizes and is more widely disseminated, we would move to measures that are more objective, more intermediate in their time frame, and reflect the perspectives of people beyond the immediate trainees. We might have trainees managers rate how frequently they observe each of the outcome behaviors in trainees over the three-months following the training. Finally, as the program demonstrates its consistency and immediate impact, we would move more to measures that are very objective, long-term, and from a perspective that is considerably removed from the trainees themselves (the ‘b’ cube in the diagram). For instance, we might have conduct a meta-evaluation to attempt to assess the annual income gain that might be attributable to each unit increase in each outcome of training. This dimensional measurement approach emphasizes that there are many varieties and flavors of measurement and that it is desirable to use a mixture of measurement types over the evaluation lifetime of a training program. But we would probably not have equal amounts of all types of measures. More likely, measures that are less rigorous and less costly would dominate the system with more rigorous and expensive measurement reserved for situations where the stakes are high or the level of training expense is great enough to warrant the expense and effort of the most rigorous measurement.

PUTTING IT ALL TOGETHER: THE CORPORATE UNIVERSITY EVALUATION TEMPLATE

One of the major advantages of the concept mapping and pattern matching approach is that it can be standardized for general use in a corporate university while at the same time allowing for flexible tailoring where the situation requires. Here, we outline the general strategy that would form the template for training evaluation, recognizing that the specific application of this template would differ slightly from one training program to another.

This template assumes that we need to develop, test, and disseminate a new training program to address some recently identified new skill area in which the corporation needs proficiency. It also assumes that there are a variety of interested stakeholder groups and that the evaluation will be conducted across an approximately 2-3 year development/implementation cycle.

Phase I. Conceptualization. The first phase of any training evaluation will be the conceptualization one. Several major tasks will be comprise this stage:

Concept Mapping of the Training Program. The key identified stakeholders will develop a map of all of the topics that should be included in the proposed training program. This map will be the initial conceptual guide to what the training will consist of, showing all the specific topics, groups or clusters of topics and their relative importance (see next).   Concept Mapping of Outcomes. Even before the training program is initially designed, key stakeholders will develop a concept map of the outcomes desired from the training. There are several reasons for doing this at this stage. First, it reinforces the idea that the reason the training is being developed is to have specific effects on trainees. From the very beginning and throughout the development phase, the design of the training program is always focused toward the outcome. Second, developing an outcome map at this early stage makes it possible to develop outcome measures that can begin to be tested and debugged as the training is going through initial pilot testing.

Phase II. Consensus. The second phase is designed to determine the degree to which different stakeholder groups are in agreement about key training components and objectives. Two key consensus assessment pattern matches will be performed:

Pattern Matching of Training Topic Importance. Each key stakeholder group will rate each of the training topics for how much relative importance they should be given in the training program. For example, typical stakeholder groups would include the key partners involved in that training area and the program developers responsible for designing the training. Pattern matching of these ratings will show whether there is a consensus across key stakeholders about what the training should emphasize. It is especially important at this early stage to identify if there are any major disagreements about what topics should be emphasized so that these can be resolved and program development can proceed with consensus.   Pattern Matching of Outcome Importance. Here, we compare the importance ratings of the outcomes for two or more key stakeholder groups. This will tell us whether there are significant differences in what each group expects will happen as a result of training. As a result of these matches, it should be apparent where there is consensus regarding the training and its expected outcomes. On the basis of these analyses, it may be desirable to redesign the training or to involve the various stakeholders in discussions designed to understand better the nature of the differences in perceptions.

Phase III. Pilot Testing and Process Evaluation. The third phase of the training evaluation will be the pilot testing one. Several major tasks will be comprise this stage:

Pattern Matching of Training Implementation. One of the most important tasks at this stage is to assure that the training is being implemented as intended. This can be accomplished by comparing topic-level measures of the implementation of the program with the intended emphasis each topic was supposed to have (obtained during the conceptualization phase). This will quickly identify topics that may be receiving too much or too little emphasis in the pilot version of the training so that appropriate revisions can be made.   Pilot Pattern Matching of the Outcomes. Even at the earliest stages of training program testing, it is important to build in standard evaluation procedures that will provide feedback about the effect the training is having. In this phase, a number of pilot outcome measures will be developed and pilot tested. These will range from immediate subjective ratings of change be trainees to some initial intermediate ratings by co-workers or managers of trainee change in performance. It is important to recognize that the measures taken at this stage will not constitute a valid evaluation because they will themselves be under development, will generally be based on smmall sample sizes, and will be reflecting a training program that is still under significant revision. Nevertheless, this pilot measurement of outcomes will be especially valuable in providing quick initial feedback that can identify any potential major problems in the training, and will enable more reliable and valid measurements to be constructed for the full training implementation phase.

Phase IV. Full Implementation and Outcome Evaluation. The final phase of any training evaluation will be the full implementation one. It is assumend that at this stage the training program will be relatively stable and that a good set of short- and middle-term measures have been developed. Several major tasks will be comprise this third and final stage:

Pattern Matching to Monitor Ongoing Implementation of the Training. The procedures that have been refined during the pilot phase to assess the implementation of the training will be applied at regular intervals during full implementation of the training. This will provide an efficient and low-cost means to assure that the quality of the training will be maintained over time.   Pattern Matching to Assess the Effect of the Training. At this point, the full effects of the training program can be assessed and monitored over time. Longer-term and more objective outcome measures will be developed and implemented in addition to the sort- and medium-term measures developed earlier. These will include, but not be limited to, measures of client satisfaction, of value added to engagements, and of ROI.

This general template provides the major sequence of activities over the life-cycle of a training program, but does not limit what can be done or how the evaluation might be tailored to address specific issues of importance.

The template offers several key advantages over procedures that are commonly used for evaluating training. Because it encompasses the entire life-cycle of the training rather than only entering the picture after the program is already finalized, the evaluation can help improve the training as it's being developed. Because it offers a general template that can be applied to all training situations, it reduces the need to redesign evaluation for each new training program and improves the overall efficiency and cost-effectiveness. Because it uses the concept mapping approach throughout, all relevant stakeholders can be efficiently included in the development process and can easily comprehend what is happening with the training and its outcomes. Because it uses a pattern matching approach, it is possible to examine the training in great detail without losing sight of the overall picture. This leads to more specific and useful feedback about the training while at the same time enabling an overall determination of its effect.

Summary Overview of Concept Mapping and Pattern Matching Template for Evaluation