William M.K. Trochim

Cornell University
and
Concept Systems Incorporated

Author's Note

Much of the material in this paper was originally included in my responses to a series of questions about corporate creativity and how it could be enhanced. The questions were posed by Ian Rose, President of IBR Consulting Services Ltd. and my answers were to be edited along with others into a single report given to a select group of corporate sponsors. After reviewing my responses, Ian felt that my discussion of concept mapping would be more coherant and useful if it was removed from the original question-and-answer structure of the report and repackaged into this stand-alone document that could be distributed along with the larger report.

I want to thank Ian for his assistance and advice on this paper, and for the opportunity he has provided me in suggesting that his readers would benefit from this document.

Concept Mapping for Organizational Creativity

How can we make our organization more creative and productive? Especially in the pressure of a world economy and intense market competition, it's essential that businesses harness all of their creative capabilities to more effectively address the challenges they face. This paper describes a technology called concept mapping that is particularly useful for enhancing group or organizational creativity. There are three main purposes in this document. First, you are introduced to the basic concept mapping technology and how it worksand how it works as a "stand-alone" process. Second, a case is made for integrating concept mapping technology into the organizational culture rather than viewing it as a "stand-alone" method. And third, an example is given that illustrates how this integrated creativity technology might be used to help your organization define what it means to be creative in your context and measure the effects of that creativity.

What is Concept Mapping?

For the past fourteen years, I have been developing a general method, called concept mapping, that is explicitly designed to help groups and organizations think more creatively and effectively as they seek to innovate. It has several critical characteristics. It helps individuals think as a group without losing the individuality that is essential for creativity. It enables groups to manage complexity without trivializing or losing important detail. Concept mapping is a process that enables a group to lay out ideas on any topic in a picture or map that can guide them in subsequent planning and innovation. The concept mapping process is:

	Participant-oriented: It allows the creativity of all of the participants to influence the final product. Broad-based organizational participation is encouraged.
	Bottom-up: It starts with concrete and specific creative ideas stated in the language of the participants and moves gradually to more general and abstract creative concepts. The underlying principle is "individual input --> group output."
	Structured: It has a definite beginning, middle, and end--it won't let a group get tied up in endless meetings or discussions.
	Simple and intuitive: It requires only that participants be able to brainstorm, sort, and rate--all fairly common, familiar activities.
	Visual: The graphical results concisely summarize the creative thinking of the group. They are readily understood, can be presented easily, and are remembered by participants long afterwards.
	Rigorous: The heart of the process uses advanced, well-established statistical approaches (although these are not directly visible to the group).
	Scaleable: The process can be used equally effectively on small or large organizational change problems. It works as well with a committee of fifteen as with groups of hundreds or even thousands.
	Transferable: Once the basic approach is embedded in an organization it can be used to address a wide range of tasks that require groups to work together creatively and effectively including strategic planning, program and product development, business reengineering, change management, requirements analysis, training development and evaluation.

Concept mapping is a general method that can be used to help any individual or group describe their ideas about some topic in a pictorial form. There are several methods that currently go by names like "concept mapping", "mental mapping" , "mind mapping", or "concept webbing." All of them are similar in that they result in a picture of someone's ideas. But the kind of concept mapping described here is different in a number of critical ways. First, it is primarily a group process and so it is especially well-suited for situations where teams or groups have to work together. Other methods work primarily with individuals and lack mechanisms for aggregating the individual's products into a group map. Second, it uses a very structured facilitated approach. Specific steps are followed by a trained facilitator in helping a group articulate its creative ideas and understand them more clearly. Third, the core of concept mapping consists of several state-of-the-art multivariate statistical methods that accomplish the key step of aggregating individual ideas into a group product. And fourth, the method takes advantage a of specialized computer program that can handle the data from this type of process and easily accomplish the correct analysis and mapping procedures.

The Concept Mapping Process

The concept mapping process involves six steps that can take place in a single day or can be spread out over weeks or months depending on the situation (see Figure 1).

The process begins with the Prepare Project step. Three things are done here. The facilitator of the mapping process works with the initiator(s) (i.e., whoever requests the process initially) to identify who the participants will be. A mapping process can have hundreds or even thousands of stakeholders participating, although typically a relatively small group of between 10 and 20 stakeholders is involved. Second, the initiator works with the stakeholders to develop the focus for the project. For instance, the group might decide to focus on defining a new product or making a decision. Or, they might choose to map all of the outcomes they might like to see as a result of some effort. Finally, the group decides on an appropriate schedule for the mapping.

Figure 1. The six steps in the concept mapping process.

In the Generate Ideas step the stakeholders develop a large set of statements that address the focus. For instance, they might generate statements that describe all of the specific options they have for addressing a problem or change in the organization. Or, they may create statements that describe specific topics they need to address in a training program. A wide variety of methods can be used to accomplish this step including traditional brainstorming, brainwriting, nominal group techniques, focus groups, qualitative text analysis, and so on. The group can generate up to 200 statements in a concept mapping project.

In the Structure Ideas step the participants do two things. First, each participant sorts the statements into piles of conceptually similar ones. Usually they do this by sorting a deck of cards that has one statement on each card. But they can also sort directly on-screen using the computer program by dragging statements into piles that they create. Each person can have as few or as many piles as desired. The participant also names each pile with a short descriptive label. Second, each participant rates each of the statements on some relevant scale. Usually, we initially rate the statements on a 1-to-5 scale, where a 1 means the statement is relatively unimportant compared to all the rest, 3 means that it is moderately important, and 5 means that it is extremely important. Later in a project, we might incorporate a wide variety of additional metrics, like the degree to which we've accomplished each item to date; the amount the item is evident in the final product; the effect on each item; and so on.

The Compute Maps step is where the analysis is done -- the process of taking the sort and rating input and "representing" it in map form. Two major statistical analyses are used. The first -- multidimensional scaling -- aggregates the sort data across all participants and develops the basic map. On this "point map" each statement is a point and statements that were piled together by more people are located closer to each other by the analysis. The second analysis -- cluster analysis -- takes the output of the multidimensional scaling (the point map) and partitions the map into groups of statements or ideas. If the statements consist of the topics that need to be covered in a training program, the clusters of ideas show how these can be organized into logical groups of topics. If the statements are specific outcomes, the clusters might be viewed as outcome areas or measurement constructs.

In the fifth step -- the Interpret Maps step -- the facilitator works with the stakeholder group to help them develop their own labels and interpretations for the various maps. This follows a well-defined series of steps that enable the group to achieve easily and effectively an understanding of their thinking.

Finally, the Use Maps step involves using the maps to help address the original focus. In a program development effort, the maps can be used as a visual framework for describing the program. In an evaluation, they can be used as the basis for developing measures and displaying results.

An Example Concept Mapping Project:

What Are the Characteristics of a Good Business Reengineering Consultant?
To give you a better idea of how concept mapping is used to encourage organizational creativity and innovation, let's consider an example project.

This project was done to identify the competencies that a consultant needs in order to do business reengineering effectively. It was seen as a first step in developing a curriculum for reengineering consultants. There were a number of important issues in addition to competency identification. For instance, reengineering consulting tends to be done in teams, with different team members playing different roles. It is likely that the competencies needed for one role may not be the same as those needed for others. It's also important to distinguish between core competencies that every consultant needs to have and reengineering-specific competencies.

Nine persons who were experienced in reengineering consulting participated in this project that lasted less than one day. They first generated 100 statements describing competencies that reengineering consultants need to have in order to be successful. Each person sorted the statements into groups and rated them on a 1-to-5 scale of relative importance. The concept map that resulted divided these competencies into ten clusters (see Figure 2).

Figure 2. Cluster rating map for project to establish competencies for business reengineering consultants.

On a concept map, the important characteristic is distance -- statements and ideas that are closer together are more similar, ones that are farther apart are less related. In this particular map, for instance, the cluster Plan, Organize and Manage Project is adjacent to the one labeled Staging for Implementation because the two ideas are related. Notice that the concept of Creative Thinking was rated very high in importance for reengineering consultants (as indicated in the greater number of layers in its shape), along with the concept Communication. When the group looked at the map, it led them to considering in a new light what they think reengineering is -- there didn't seem to be anything (at least on this high-level map) that was specific or unique to business reengineering consulting, as opposed to general management consulting. They questioned their fundamental idea of reengineering itself -- does it call for a distinct or unique set of consulting skills, or does it simply require more intensive application of traditional consulting skills?

The participants were also concerned that while the importance ratings reflected what competencies were important overall, they didn't indicate how competencies differ across different team roles. They identified three major roles on a typical business reengineering consulting team: the Project Leader, the Team Leader and the Team Member (this model assumes multiple teams working together on a single reengineering project). The participants were divided into three groups of three persons each and asked to rate each competency statement for how important it is for each of the three roles (therefore, each rating has an N=3).

To explore the results of this rating pictorially, we use a technique called pattern matching. Pattern matching is used whenever we want to compare two ratings or measures that were performed for the same concept map. We can use it to look at the consensus among different groups (e.g., managers and staff) to look at the consistency of a process (e.g., over time in a quality assurance framework) or to assess outcomes (e.g., to look at the "return on expectations" of a training program). The results are graphed in the form of a ladder graph. The ladder graph shows one line for each cluster on the concept map. Where the line intersects each vertical axis is where the group for that axis rated the cluster. Horizontal or flat lines indicate a similar ordering or rating; sharply angled lines suggest that there is a "disconnect" between the two ratings. Every pattern match is also accompanied by a correlation that estimates the overall degree of agreement between the left and right axes.

Figure 3. Match between competencies needed for Project Leader and Team Member in a business reengineering consulting project.

The pattern match between the two leadership positions (Project and Team) was fairly strong (r = .78), suggesting that the competencies needed were similar across both roles. However, the pattern match between either of the leader roles and the Team Member role was negative, with the strongest negative match between the Project Leader and Team Member (r = -.47). This last match is shown in Figure 3.

It's clear from the ladder graph that the competencies needed differ considerably by role, with the Project and Team Leaders having similar competency needs while the Team Members differ considerably. Closer examination of the pattern match indicates that while the top competency area for the leaders is to plan, organize and manage the project, that was the lowest competency area for the team members. On the other hand, team members needed to be creative and know the tools and analytic methods necessary for reengineering. One implication of this result is that it is probably not necessary to develop three different tracks in the training curriculum, one for each role. Instead, two tracks would be sufficient, one for the leaders and one for the members.

Figure 4. Pattern match of competency importance with whether the competencies are reengineering-specific.

The other question of interest was which competencies are specific to reengineering and which are more generic core consulting competencies. The nine participants rated each of 100 statements dichotomously where 0=core and 1=reengineering-specific. Fewer than 30 of the 100 statements were rated by five or more people as being reengineering specific, suggesting that many of the competencies are generic ones. Furthermore, a pattern match (see Figure 4) of overall importance with reengineering-specificity (r = -.24) suggested that the more important competencies were generally not the reengineering-specific ones.

For instance, the competency deemed most important -- communications -- was one that was perceived as a core competency rather than a reengineering-specific one. One important implication of this for training is that specific reengineering training may be able to be quite targeted and limited, as long as one can assume that the trainees are already very competent in core consulting skills.

This is just one example of a concept mapping project. It was accomplished in less than one day by only nine participants. It formed the basis for subsequent development of a training curriculum for business reengineers. It enabled each of the participants to contribute creatively to the effort. It channeled this creativity in a structured way so that the group did not lose focus. It led to considerable insight about the nature of reengineering in a way that enabled the group to suggest that reengineering training could be targeted to specific topics and had to assume consultants already had strong general consulting skills. The result of this channeled, creative effort was an innovation -- a different type of curriculum than was originally envisioned by any of the individual participants, one that was able to be smaller, better targeted and less costly.

An Integrated Creativity Methodology

Concept mapping is a tool. Its utility in practice depends on how well it can address real-world problems. The initial tendency, with any tool, is to wait before implementing it until the appropriate circumstances for using the tool arise. The problem with that approach is that it is entirely reactive rather than proactive. It encourages the user to think of the tool as a "one-off" experience, something you pull off the shelf for those occasions where it seems to fit. In most organizations this is a clear recipe for failure. In the chaos and haste of daily events, even when an appropriate opportunity arises, the organization will not be ready to deliver the technological capability in a timely and efficient manner. The method always has to be pulled off the shelf and "dusted off" before it can be used. Facilitators are brought in at the last minute. Participants have to be "re-sold" on the idea each time.

I want to suggest a better approach, an integrated, proactive strategy for continuous organizational creativity. This approach is based on the idea of systems integration -- that creativity technologies have to be well-integrated into the standard operating procedures of the organization to achieve the benefits of creativity. This requires that we analyze the systems we currently use to achieve creativity (however informal these may be), identify where concept mapping technology can be usefully integrated, and provide the training and support to make the technology an integral part of standard organizational operating procedures.

What would an integrated methodology for creativity and innovation look like? Based on our work in the area of corporate training and development with a number of client organizations (Andersen Consulting Education, Amtrak, Price Waterhouse, EDS) I'll describe the general process that is followed (rather than detailing how each organization implements it) so that it will be more apparent how one might integrate such methods into other contexts.

The method is based on a project "life-cycle" approach. We define four steps or stages that would be present in almost any project or process. We would almost certainly have some variation of this set of steps in any innovation development effort. These steps would be usefully applied for training or product development, strategic or operational planning, change management or business reengineering, information systems integration, client satisfaction analysis, marketing plan development, and so on. The steps are:

Conceptualization. The process of involving the creativity of stakeholders in the development of a shared vision of the project, exploring consensus and areas of disagreement, organizing into teams or task groups, and conducting relevant needs assessment.

Development. The process of working as teams within a clearly mapped shared vision to address the project goals (e.g., develop the training program or product, strategic plan, requirements for an information system, specific reengineering plan for a company or organization, change management strategy, and so on).

Implementation. The process of pilot and field testing the innovation, including the implementation in a range of contexts and conditions, and the assessment of the degree to which the innovation faithfully represents the shared vision guiding the project.

Evaluation. The process of assessing immediate and longer term consequences of the innovation with the dual purpose of continual improvement and overall estimation of impact.

Of course, the process is not as linear as this four-step sequence suggests. There are continual opportunities for feedback and reconsideration along the way. And, it's assumed that this sequence would be undertaken within a larger strategic organization plan so that projects that reach the evaluation stage provide information to the organization that is used in setting the course for subsequent innovation projects.

Given an overall process for innovation along these lines, the organization needs to develop the methodology to support the effort. This methodology should be detailed, flexible, rigorous, usable, and well-supported. To give you an idea of how this works, I'll describe some of the details of how each step are addessed in the general methodology when applied to a business training development context.

Conceptualization. A project typically begins with an Initiator, someone who sees the need for new training or support in a specific context. The Initiator works in the organization to get approval for the concept and the go-ahead to begin examining the feasibility of developing a training program. Usually, a small team is assembled to begin the effort. Their jobs are to define a focus for the project, identify all of the key stakeholders (e.g., the Initiator, other potential clients, content experts, potential development team members, relevant managers and administrators, potential training participants, and potential trainers), and develop an initial schedule for the conceptualization effort.

The next stage is to "map" the potential content of the training and begin to explore the potential shared vision of the various stakeholder groups. The mapping begins with the listing of all of the potential topics that might be included in the training. This is usually accomplished through some combination of brainstorming, solicitation of statements via e-mail, nominal group process, and so on. The statement set is edited, often through several iterations, with feedback at each step from the stakeholder groups. At some point, the various groups reach consensus about the initial set of ideas or statements, often coming up with over one hundred individual items.

The process of generating the statement set, or content domain, is inherently a creative one that requires input from a wide range of participants. But, the Facilitator of the process is responsible for keeping the group task-oriented with the goal of arriving upon an agreed-upon statement set. This entire statement generation phase can be accomplished in a short workshop or, if participants cannot easily be assembled, can be done over several days via mail or computer.

Once the content domain is identified, the participants (comprised of all relevant stakeholders or of a selected subsample) structure the domain. To accomplish this, they do two simple tasks. Each person sorts the ideas in the content domain into piles of similar ones. This "unstructured" sort allows each person to organize the ideas as they see them. Some people will sort the statements into lots of detailed piles; others will "clump" them into only a few. The participant is free to organize them in a way that makes sense to them individually. In addition, each person is asked to rate the statements on one or more relevant dimensions. Usually, at this early conceptualization phase, the participants would rate the statements for their relative importance or emphasis in the training. The structuring (sorting and rating) can be accomplished easily in a number of ways. Participants can be sent a packet of sorting cards and rating sheets and asked to do this manually. Or, there is a computer program that allows them to do the sorting and rating on-line.

When the structuring information is gathered, it is entered into the computer and the map analysis is done. The analysis takes the individual input from the participants and generates group maps that reflect this input. The Facilitator has the job of assembling the various maps that are developed, and preparing for the interpretation by the participant stakeholders.

The interpretation task can be accomplished at a structured meeting of all participants, or a subgroup of the stakeholders may be sampled. In the interpretation, the facilitator leads the group through the different maps that were developed, helping them to understand how their ideas have been portrayed, how they are interrelated, and where there appears to be consensus about what's important. The interpretation session gets the group to a "map" of the training program. The map describes each of the broad topics that need to be included in the training and their relative importance. For instance, the final map that was developed for a course in computer systems design is shown in Figure 5.

Figure 5. Final map for a computer systems design course.

Each shape or cluster on the map shows the general topical area for the course. The heights or layers for each cluster show the average importance rating the cluster was given by the participants. Topics that are closer on the map are generally more interrelated than ones that are further apart.

In this map, we see that the four topics across the top of the map are related to the specific programming skills to be taught in the course. Along the bottom, we see topics related to the management of a systems design effort and to the personal and team attributes that are relevant. It's interesting to note that the Client Issues and Change Control topics fall into an area of the map that "bridges" between the technical computer skills on the top and the human and management skills on the bottom. This map shows the final product of the interpretation. But there is considerably more detail than is shown in this final map alone. For instance, each topic has within it a detailed set of statements that describe the specifics of the topic. For instance, the Management topic on the lower left has the following statements:

	Manage time effectively.
	Manage resources effectively.
	Scan a multitude of information and decide what is important.
	Decide how to manage multiple tasks.
	Organize the work when directions are not specific.
	Work quickly and effectively under pressure.

It is critical in the interpretation stage that the various stakeholders begin to develop a shared vision of the project. The map provides them with a potential vision, but they additionally need to examine the degree of consensus that exists and to identify any major misunderstandings or "disconnects" if they are going to move successfully into implementation. They accomplish this by examining the degree to which their views of the map content match up. We use the pattern matching method for depicting this match.

Figure 6. Pattern match between the importance ratings for faculty and developers on the key topics for the computer systems design course.

For example, Figure 6 shows a pattern match between two key stakeholder groups for the computer systems design course. On the left side of the "ladder graph" we see the average importance ratings for each training topic for the faculty. The faculty, in this case, are virtually the same as the initiator -- they are experienced systems designers brought in from the field to teach the course. On the right side we see the average importance ratings for the course development team. In order for them to be able to do an effective job of "translating" the vision of the practitioner-client into a well-designed course, they must have a clear understanding of what their client thinks is important.

The figure shows that there is only moderate agreement between the two stakeholder groups. The practice-oriented faculty members place the greatest emphasis on the human side of the endeavor -- the client and team issues, and the personal awareness and skills of the team members. The developers tend to weight more heavily the technical skill of developing the graphical user interface and the personal awareness and skills of the members of the team. Clearly, the biggest "disconnect" is on the topic of documentation. Faculty see it as the least important topic while developers give it considerably more weight. This makes intuitive sense because the faculty know that in practice documentation is often hard to fit in well given typical resource restraints. The developers, who probably approach the content from a more theoretical and less experienced framework believe that documentation should be considered more important than it typically is. Which group is "right"? Clearly, that's a difficult question. Practitioners would probably agree that they should spend more time documenting their work, but would argue that they typically don't have the time or resources. Developers would probably acknowledge the practical difficulties but argue that good documentation is critical to a high-quality product. In fact, it almost doesn't matter who is "right" here. At some level it's easy to see and agree with both sides of the argument. What is critically important, is that these kinds of differences surface early in the life of the project. Stakeholders need to know where there is the potential for disagreement. They may decide in the end to agree to disagree on this issue, but at least they should consider the potential disconnect and discuss its implications. If they don't, the organization is likely to devote considerable resources to developing a course that faculty will see as "theoretical" and "unrealistic" and developers will argue is not being implemented correctly because of the 'fine tuning" of course delivery by the faculty who are neglecting the material on documentation. In cases like this, the organization should not proceed to development until they either have a stronger consensus between stakeholders or the groups agree to disagree and state why. At the least, it's important for everyone involved to understand how they view the product that's about to be developed.

Notice that throughout this example, the methodology is seamlessly integrated into the work that needs to be done. Once the organization sets up the methodology and the normative expectation that it will be used, the various stakeholders simply consider this the way they do their business. It becomes unthinkable that you would begin developing a training program without first mapping out the content. Everyone looks for the comparisons about consensus to make sure that there is some shared vision about what is to be worked on. They aren't doing 'concept mapping" or "pattern matching" as a stand-alone activity -- they are taking the first steps in developing a course.

Development. At this point, the process begins to move into the development phase. In many organizational contexts, there would be a great potential for disconnection as we move from one phase to another. A different group of people typically become involved in the project here (in this case, course developers) and they may or may not understand well the nature of the task at hand. But in this process, they have been there from the beginning. They were involved in the conceptualization of the training. Their views were contrasted with other key stakeholders and discrepancies were addressed.

As the development team works, they use a number of methodologies borrowed from training development to translate the content of the map into a course. For instance, they might review each topic on the map and examine how it might best be delivered (e.g., through lectures, role playing, discussion, interactive multimedia, etc.). The Facilitator would help them make rapid assessments of delivery mechanisms display them on the map. In a large development effort, the team will probably have to break down into subgroups to address different content areas. Again, they would use the map as a tool for displaying who is working on which topics at any given time. When they issue periodic reports about the progress of the training development, they would use the map as a visual device to show what's being addressed, in what ways, and by whom. The development team can also assess, on a regular basis, how much progress they are making in addressing each of the specific course topics (i.e., the statements on the map) and broader topical areas (i.e., the cluster areas). The map becomes a public mechanism for charting their progress. One would expect to see the original map posted throughout the development team offices, providing a visual indicator of progress.

In this way, the organization leverages the original creative effort, getting more value out of it than they would if they only used the maps during the conceptualization phase. They are helping to keep the project "on track" with respect to the shared vision. And, the development team is able to be accountable for their work in a manner that other stakeholders will readily understand, because reporting will be in terms of a mutually developed shared vision.

Implementation. At some point, the project will be far enough along that it will become important to begin pilot testing implementation. Initially, the development team will bring in key stakeholders to do "user tests" -- to examine the course as developed to date and to react to how well it's addressing the original shared vision. The map provides a framework for assessing the fidelity of the translation of the original shared vision into an operational training program. Stakeholders might be asked, for instance, to do a quick assessment of how well the training program addresses each of the content topics. This can be "matched" with the original importance ratings to help assure that the topics that appear to receive the most emphasis are the important ones. The map and match graphs can be examined and posted as an ongoing indicator of progress. This information is useful to the developers because it shows overall how well they're reproducing the original vision (e.g., through the overall "grade" of the pattern matching correlation), and it provides specific and detailed feedback about which areas may not be receiving enough emphasis.

When the program is nearly complete, it is standard procedure to do several pilot implementations. Here, it's desirable to assess the degree to which the training as implemented delivers the desired content. It's also critical that we show that the training can be delivered consistently over time. Again, we can use the original map of the content as a framework for assessing and displaying this information.

Figure 7. Consistency pattern matches across three pilot and two actual administrations of the client/server computer application development training program.

For instance, Figure 7 shows a consistency match for the initial implementations of a training program that teaches client/server application development. Each line represents a content area on the original concept map for the project. The vertical values at which the lines hit the axes show the self-reported performance of training participants for the last three pilot implementations and the first two real conducts. The figure demonstrates that the training is consistently getting a similar pattern of results -- the effects on student performance appear to be remarkably stable.

In most organizations, the implementation phase is accomplished by different groups than were involved in the development phase. Where development focused around development team members, the implementation revolves around faculty and students. The development team is gradually "disengaging" -- moving on to new projects -- as the faculty and training managers begin taking over the delivery. As in all functional shifts, the organization faces the danger that there will be a significant disconnect, much like the runners in a relay race who fail to pass the baton successfully. But in this integrated methodology, the faculty and representatives of the target students were involved from the very beginning -- they took part in the development of the map. Their understanding of the project was contrasted with that of other stakeholder groups. And, the implementation results are presented using the same map that was developed initially and that provides the shared vision for the project. In fact, the participants might not even think of this implementation step as something distinct from or standing apart from earlier and later phases. It's just the way they do their business, the way they develop and deliver training.

Evaluation. At some point, the training is being implemented on a regular basis and we will want to assess what effects it is having. The effects can range from the most immediate knowledge or attitude shifts to longer-term impacts on the organization, its clients, and its bottom line. A key issue is whether the program is accomplishing what it was intended to accomplish. The approach taken here is to look at the "Return On Expectations" -- the degree to which the program is accomplishing what the stakeholders set out in the original shared vision.

Figure 8. Achievement outcome pattern match for an Introductory Computer Systems Analysis course.

Figure 8 shows an outcome match that compares the original importance ratings for each topic against a measure of effect, in this case participant self-reported achievement (one could substitute as outcomes any types of measures including objective tests, manager ratings, client satisfaction, and others, as long as these were structured so that they could be linked to the topics on the course content map). On the left side, we see the importance rating for each topic. On the right, we see the outcome measure. The first thing to notice is that there is a moderately strong relationship between the two. The correlation of .68 indicates that, in general, the expected emphases are evidenced in participant outcomes. But the pattern also gives us useful diagnostic information for revising or improving the training. For instance, one of the largest "disconnects" is on the Client Issues topic which was thought to be most important but is only moderately being affected (relative to some of the other topics). Although we could probably say that on the whole the program is achieving desired outcomes, the program might be improved by redesigning the client issue portions to make them more salient.

As in all the other phases, these efforts are seamlessly integrated into the training development process. The organization doesn't wait until this phase to call in an evaluation team. That team has been there from the original conceptualization phase (in fact, it is often the evaluators who facilitate the entire process). And, when the outcome matches are shown to the various stakeholder groups, they don't see them as entirely new or unusual figures. They know the content well because they have been involved throughout the project life-cycle. They've seen a number of similar graphs and maps that addressed the conceptual, development and implementation issues. The evaluation doesn't appear to be a distinctly different phase -- it's just the continuation of how they do their training development business.

Practicing What We Preach: Using Creativity Technology to Measure Creativity

In organizations that attempt to enhance creativity, eventually the question of effectiveness will arise: What are the effects of our creativity efforts? Measuring effectiveness of efforts to enhance something as complex and ephemeral as creativity poses a significant evaluation challenge, one that requires considerable measurement creativity. How do we define creativity? How does it manifest itself in our organization? If you have an ongoing integrated methodology that enhances creativity like the concept mapping approach outlined above, it makes sense to turn that methodology on itself and use it to help define what creativity means and how it might be measured. The results will define creativity from your own perspective and in your own organizational culture. The measurement system you could construct based on such a study would be your own, stated in terms your organization understands, and tailored to your needs and situation. And, there's an added bonus: to do this well, you would involve a range of stakeholders throughout your organization in a participatory, efficient, systematic creativity effort while at the same time trying to measure creativity! This section provides an outline of how I would do this.

We might begin with a small team of stakeholders (perhaps five people) from throughout your organization to develop the focus for the project. In this case, it shouldn't take more than one hour to come to consensus on a focus statement. It might be something like this:

Generate statements (short phrases or sentences) that describe specific things we could observe that would indicate that creativity is occurring in our organization.

The focus is stated in the form of a command because this is the exact instruction we want to give people for generating statements to make up our concept domain. We also want this instruction to be specific and concrete. If you ask most people to come up with indicators of creativity you'll draw a blank because most of us don't think in those kind of abstract measurement terms. But if we focus on "specific things we could observe" most of us could readily generate a short list without too much trouble.

The next step would be to generate that list, the statements that constitute the content domain. We might hold several brainstorming sessions with a variety of people in our organization (it might even be a good idea to include some of our representative customers, suppliers, partners, and so on). These would be short meetings, no more than an hour each. For those who can't attend these "focus" sessions we'll offer the option of using the company e-mail system to generate items. We'll have a small editorial team go through the initial set and edit them for redundancies, preserving the original language and detail as much as possible. Throughout the editing process, we'll ask the initial project team of stakeholders to review the current edited set and tell us what might be missing.

Next, we need to organize all these statements. We'll involve a small and varied group of our stakeholders, each of whom will be asked to sort the statement set into piles of similar statements. Fifty people at most would suffice. Each would spend less than an hour. In addition, we'll send out a survey to everyone in the organization asking them to rate each statement on a 1-to-5 scale where 1 means the statement is relatively unimportant to the creativity of the organization, 2 = somewhat important, 3 = moderately important, 4 = very important, and 5 = extremely important. We send this to everyone for several reasons. We want the broadest representation possible. It's a relatively inexpensive measure to send and complete. And, we will want to compare different subgroups later (e.g., managers versus staff). Each respondent will need perhaps a half hour to complete this simple rating.

Now, we enter the data into the computer. The amount of time required depends on the size of your organization, but it depends even more on how you gathered the data. We have the technology to have people sort and/or rate the statements visually on a computer. If we use that method to collect the data there is obviously no data entry cost. It would take less than an hour or two to run the analysis and produce preliminary maps of the ideas of the group. We would draw one map that shows the individual statements. We'd do another that shows how the statements are grouped into 'clusters" or types of creativity indicators.

At this point, we'd engage the stakeholders in interpreting the maps. This could be as complex or as simple as we desire. At the simpler end, we might hold a two hour meeting with a representative group of twenty or so stakeholders. If we wanted to engage a wider audience, we might hold multiple sessions to have different stakeholder groups interpret the same group map. As a compromise we might hold two or three independent sessions so that we could convince ourselves that there is consistency in interpretation across independent groups. At each interpretation session, we would begin by showing the participants the statement map, taking them on a "tour" to orient them to how the statements are arranged. They would see that statements that are similar are closer on the map and ones that are less related are farther apart. They would begin to recognize that certain areas on the map are identified with certain types of statements. We would then show them how the statements on the map were partitioned into clusters in the analysis. They would examine the statements in each group and suggest a short phrase to identify the group. The computer would suggest phrases that fit the cluster well that are taken directly from the way they labeled their statement piles when they individually sorted. We would reach agreement about the labels for the clusters of creativity indicators. At this point, we would have a map that might look something like the one in Figure 9.

Each cluster on the map represents a collection of specific indicators of creativity. On the top of the map we find two clusters that include indicators of individual creativity. The cluster Personal Characteristics of Creativity might include statements like: ability to offer constructive suggestions, ability to think on your feet, or generates lots of "out of the box" ideas. on the lower right of the map we have the indicators of organizational creativity. One cluster deals with the strategic creativity of the organization -- its ability to respond to major market shifts or competitive threats. One deals with the flexibility of the organization and might include specific indicators like: able to offer employees flexible working schedules, willingness to experiment with work-at-home models, or new ways of delivering training. At the bottom of the map is a cluster related to creativity in team situations. Here we might see statements like: able to mediate well on team projects, encourages effective group process, or generates new solutions for team to consider. As we move to the left on the map we deal with areas of creativity that affect how the organization interacts with the outside world. One cluster relates directly to relations with clients, the other is more directed towards creative presentation of the organization's image (marketing and public relations creativity). For Approaches to the Client we might find statements like: able to generate new sales opportunities, able to work well with "cold" leads, or ability to address client complaints effectively.

Figure 9. Hypothetical Map of Creativity Indicators for an Organization

At this point we could use the map in a number of ways. First, we might overlay the average importance ratings for the entire group. This would give us a visual overview of which areas of creativity are thought to be relatively more important. This might give us a picture that looks like Figure 10. The clusters judged most important (across all stakeholder groups) are the Personal Characteristics of Creativity, Strategic Creativity, and Organizational Responsiveness & Adaptability. The external environment clusters on the left were relatively low in importance. Perhaps we would be surprised by the relatively low rating for the Creative Teamwork cluster. This might provoke discussion about whether that rating is "right" or "desirable" or representative of what the organization might want.

Figure 10. Hypothetical Map of Creativity Indicators for an Organization with Average Importance as layers.

We would certainly want to look at the consensus across key stakeholder groups. If we look at the match between "Managers" and "Staff" we might find something like Figure 11. The creativity cluster indicator labels are shown on the right of the figure in the order the staff rated them. The first thing to note is that there is virtually no relationship between managers and staff (r = -.1). That is clearly confirmed in the visual pattern. About the only thing these stakeholder groups agree on is the relatively low importance attributed to The Creative Organizational Image. There are some especially noteworthy disconnects. The top three clusters for the managers are Creative Efficiency, Organizational Responsiveness & Adaptability, and Creative Teamwork. Two of these, Creative Efficiency and Creative Teamwork, are among the lowest for the staff. On the other hand, staff clearly perceive the "personal" creativity indicators as important while managers rated them considerably lower.

What would we make of results like these? Perhaps the best answer is another question: "What would your organization make of results like these if they were yours?" At the least, when the organization thinks about what it means by "creativity" these results suggest that key stakeholder groups may weight different indicators very differently. This knowledge would, in and of itself, be useful. But it may be that discussing these results would lead both groups to reconsider their assumptions. Perhaps the managers would understand better why the staff emphasizes personal attributes more than some of the other areas. Perhaps staff could get greater insight into the possibilities for creative work on teams.

Figure 11. Hypothetical Match for Creativity Indicators for an Organization between Importance ratings of Managers and Staff.

What has become of our attempt to develop a comprehensive system to measure creativity in our organization? The next steps of the interpretation meeting would address that. We might turn the map over to the evaluation department and ask them to review the clusters and the statements within them and generate some proposed measures for each. Or, we might use the map as a structure for organizing the stakeholders into task groups (that would have evaluation specialists on each group) and charge them with developing draft indicators for each of the statements in their task group's cluster.

When we begin to collect measures we can use the same map as a device for reporting the results. Six months later we could produce a map where the layers are the initial average results in each cluster. When we show the results to various stakeholder groups, they would probably remember the original map well, may recall specific statements they added in, and would get the sense of continuity and follow-up that will make them feel that their initial work was taken seriously and utilized. One can easily envision using the map on an ongoing basis to report changes in creativity indicators over time, especially in response to concrete organizational efforts to improve creativity.

It's worth noting that this hypothetical example wouldn't be difficult to accomplish. And, depending on how it was implemented, it wouldn't take very long or cost very much. Perhaps it would be easier to just pull some indicators off the shelf. But I doubt that it would be nearly as creative, exciting, empowering or effective for the company.