William
M.K. Trochim
Concept Systems Incorporated
Introduction
When implementing human performance improvement, most organizations hope and expect that it will have an effect on the "bottom line" – that there will be a financial benefit that justifies the improvement effort. But human performance is a complex entity, and translating changes in performance into quantitative and financial results is often a daunting task. In the ideal, it is desirable to generate a causal chain of evidence from the intervention to the final financial impact.
For instance, consider a simple performance improvement intervention such as a training program. In order for the program to affect the financial bottom line of the organization, we must first assure that the training is in an area that is relevant to the bottom line. It is, after all, possible to do training on topics that are irrelevant to financial performance. Assuming that the training is relevant, we might expect that it first needs to affect the knowledge and skills of the learners. Even if it does, it will not be translated into human performance unless the learner is motivated to use the knowledge. Even if the learner wants to use the new knowledge, there are any number of factors that can prevent them from doing so, or cause them to try under less than optimal conditions. Even if the learner performs perfectly, this performance may not affect the overall performance of the business (e.g., how efficiently departments process products). And, even if there is an effect on business performance, there may not be a corresponding financial impact (depending on how relevant the business performance is to financial results). We see that in most performance improvement contexts, the causal chain from the program to final result is often a long and difficult one.
Add to this complexity the idea that organizations don’t exist in a vacuum. At each step in the causal chain there are any number of external factors that can mask or reduce the impact of a performance improvement effort. For instance, it may be that the performance program is successful but that market conditions (e.g., level of competition) leads to overall financial losses for the period in question. It may be that without the performance improvement effort, the losses would have been even greater. All of these issues point to the difficulties inherent in attempting to answer the following critical question:
There are several general approaches to estimating the business and financial impacts of performance interventions:
- Cost accounting approaches.
- Experimental approaches.
- Statistical Estimation approaches.
Cost accounting approaches are the most straightforward and desirable, but they are extremely expensive to implement. Essentially they require that we attach a dollar value to each aspect of human performance. If the performance is simple (e.g., number of pieces of product produced per hour), a cost accounting approach may be feasible. But in today’s knowledge economy many of our performance improvement efforts will be designed to affect behavior that is less easily measured. This means that we may need to try to translate subjective factors such as a manager’s ability to motivate the workforce into dollar benefits. In such common situations, cost accounting becomes a more subjective effort and loses much of its value as a method for estimating financial outcomes.
Experimental approaches involve any procedures where we can implement both a performance improvement program and observe a comparable organization or unit that does not receive the program (i.e., the "control" group). This kind of situation is also extremely difficult to implement in most organizations. It becomes especially problematic when trying to estimate the "bottom line" contributions of a program. Since financial benefits (e.g., gross income) typically occurs for the entire organization, there is effectively no easy way to obtain a control estimate of what the organization would have earned without the program.
Given the difficulties associated with both the cost accounting and experimental approaches, we are typically forced to use some type of statistical estimation procedures to estimate financial effects. There are a wide variety of such methods, but they all share one common characteristic: they use key stakeholders within the organization to estimate dollar values associated with different levels of human performance. For instance, in one common method, stakeholders estimate what the performance of a poor, average, and outstanding employee in a specific role is worth to the organization in dollar value. Then, comprehensive job descriptions for that role are developed along with specific measures for performance. Next, performance is measured and employees are scaled based on how well they perform. Finally, dollar estimates can be computed and aggregated across all employees in that role. This method, while appealing in its simplicity, is seldom feasible for estimating the effects of performance interventions. It requires that each job role affected is delineated and measured. To accomplish this approach, we typically use the salaries of employees as a proxy variable for their dollar value to the company. If managers get paid, on average, $50,000 per year, this approach assumes that their estimated value to the company for average performance is that amount. Finally, this approach provides no information about which aspects of performance were most affected by the intervention.
Estimated Financial Returns in Performance Measurement Contexts
The method described in this paper falls into the class of statistical estimation approaches to financial returns. It has several key advantages over other methods of estimating financial returns:
| It requires only a small investment of client participant time – typically less than one hour – to determine reasonable estimates of project-level financial benefits. | |
| It calculates boundaries on financial return estimates (i.e., lower and upper limits), rather than just a single value. | |
| It integrates financial return estimation with human performance measurement at all levels. |
In this approach, project costs are estimated using traditional accounting procedures. Project-level financial benefits are estimated by a client participant group using an iterative Delphi methodology. These cost and benefit estimates are proportionally distributed across performance goals and objectives and weighted by observed performance. The performance-weighted financial returns (i.e., Benefit/cost ratio and ROI) can then be presented for each performance objective, performance goal, or the whole project.
There are several key assumptions in this approach:
| Because all financial estimation methods are fallible, it makes more sense to estimate a range of financial return values within which the true value is likely to fall. In statistical terminology, rather than doing a point estimate, it is desirable to do an interval estimate. Following common statistical practice, for each financial return estimate, the 95% confidence interval will be calculated. With this interval, the odds are 95 out of 100 that the true estimate falls within the range. | |
| All financial estimates are calculated for a fixed period of time. Typically, returns are estimated on an annual basis. However, for many performance interventions, it is reasonable to expect that the major effects will accrue over time periods longer than one year. If this is the case, it will usually be desirable to estimate the returns for multiple years. Since the costs of interventions are not likely to be distributed evenly over time, it is also necessary to estimate costs for the same time periods. Depending on the situation, it may be reasonable to amortize some of the first year costs over a several year period. |
The method described here involves the following steps:
- Define the performance domain for the intervention.
- Develop a hierarchical structure from performance goals à performance objectives à performance measures
- Weight each performance objective for its relative contribution to overall benefit.
- Calculate the total costs of the performance intervention.
- Estimate the total benefit (and range) in dollar value, assuming perfect performance subsequent to the intervention.
- Proportionally distribute the total cost and benefit across the performance goals and objectives based on the relative weights.
- Measure performance on a standardized scale.
- Estimate financial returns (and range) weighted by performance for all performance objectives and goals.
Each of these is discussed in detail below.
1. Define the Performance Domain.
In this context, a performance intervention can be anything from a specific program (e.g., a training or mentoring program) within a specific department or organizational unit to an enterprise-wide performance enhancement endeavor (e.g., business reengineering, change management).
It is critical that the full performance domain that is likely to be affected by the intervention be described. (As will be shown below, the project costs and benefits are distributed across the performance domain. If key performance areas are omitted, estimates of financial returns in specific performance areas will be distorted.) It is desirable to use a structured method with stakeholder representatives throughout the client organization to delineate the performance domain.
We believe that The Concept Systemâ process represents the best approach for thoroughly describing the performance domain for an intervention. It involves a group of client stakeholders in brainstorming, organizing and mapping the performance that may be influenced by an intervention. It enables the group to identify areas of consensus and disagreement about performance objectives. It provides a hierarchical, mapped framework that represents the group’s views about performance. And, it accomplishes this quickly (e.g., a maximum of four hours of participant time) using rigorous analytical tools.
To illustrate how The Concept System helps to map a performance domain, the results from a pilot performance management project (called the GHPD project) are presented. A group of stakeholders was convened to map the performance domain for a research project. They brainstormed 80 statements that represented performance objectives. For example, some of the statements they generated were:
Develop consistent human performance messages for the marketplace Develop a common vocabulary around human performance Secure appropriate sponsorship Synergize with other measurement initiatives going on in the firm Ensure that the business development methods include human performance architecture and measurement Secure more resources on this team who can help deliver the capability to the practice Embed this capability within training Embed this capability within recruiting Continually monitor the external advances in human performance tools and methods to ensure we are on the leading edge Explicitly define the goals of each project Build the knowledge management processes to capture knowledge capital Build and deploy a human performance practice aid Link Human Performance to business vernacular Each participant then organized the 80 performance objectives by sorting them into groups of similar ones and rating them for their relative importance. The Concept Systemã software was used to develop a map of the 80 objectives. This map is shown in Figure 1.
Figure 1. Map of 80 performance objectives.
The key feature of this map is that objectives that are more similar to each other are located closer together on the map. The software then groups the objectives into higher-level clusters or performance goal areas. This grouping is shown in Figure 2.
Figure 2. Grouping of the 80 performance objectives into 9 clusters of performance goal areas.
The participants discuss each cluster of objectives and, with the aid of the software, select labels for each goal area. This labeled map is shown in Figure 3.
Figure 3. Performance Goal Areas for the GHPD pilot project.
The Performance Goal Area map is a high-level view of the performance areas that the stakeholders believe the project could influence. It is essentially a strategic level map that describes the stakeholder’s implicit idea of how the program affects performance. As is often the case with such maps, the participants discerned a pattern of performance effects that begins with the GHPD Program Strategy cluster in the lower center of the map and proceeds counter-clockwise. We might explain the pattern as follows (goal area names are italicized):
The most immediate effects of the project will be on internal project functioning – the development of our overall GHPD Program Strategy, composition of the GHPD Program Team and specific GHPD Program Planning. Assuming we perform well in these areas, we would expect to see effects throughout the firm as we roll out the project results. In order to achieve these internal effects, we need to first get initial AC Buy-In, then we need to do the Internal Selling throughout the firm, and, finally, we need to do the detailed integration into existing practices and methods, the Firm Integration. Assuming we are able to achieve these internal firm-level performance goals, we would next expect to see external performance results. First, we would detect efforts at External Selling on client engagements, followed by External Success on such engagements. Finally, we would like to capture these successes as part of our ongoing organizational learning and store them in our Knowledge Management system.
The performance map tells a "story" of how the key stakeholders envision the project affecting the entire performance domain. The story might be depicted graphically as shown in Figure 4.
Figure 4. Graphic depiction of the "performance story" for the GHPD project.
Once the goal area map is developed it can be used as a framework to display any measure of goals and objectives. Figure 5 shows how the entire group of stakeholders rated the relative importance of the different goal areas. The figure clearly shows that there are two areas – Internal Project Performance and External Performance – that were judged especially important compared with the rest.
Figure 5. Performance Goal Area map showing the average importance of each goal area for the entire stakeholder group. Goal areas with more "layers" are judged more important.
Different stakeholders in an organization often have different views of the importance of various performance goals. The Concept System software also allows the participants to examine consensus across stakeholder groups on the goals and objectives. For example, Figure 6 shows the "pattern match" consensus analysis that compares Partners and Associate Partners who have project oversight responsibility with the Managers and Consultants who will actually be responsible for carrying out the project.
Figure 6. Consensus pattern match of Partners & Associate Partners with Managers & Consultants.
The match clearly shows some major "disconnects" – differences between the stakeholders about what areas of performance are important. In this case, the disconnects correspond well to the expected preferences for each stakeholder group. The senior sponsors (Partners and Associate Partners) are primarily concerned with the ultimate external success the project will have. The managers and consultants who will have to execute the project are primarily concerned with the more immediate project needs for developing the program strategy, team composition and specific program plans. Which group is right? In fact, probably both groups are "right" considering their roles on the project. The pattern match sensitized both groups to the views of the other. Senior sponsors recognized that the desired external success will not be achieved without strong performance in the other goal areas. Managers and consultants were reminded that, no matter how well they perform on the project, it will not be a "success" unless they achieve good performance on the external successes. Even areas that are not rated highly cannot be ignored once they are on the performance map. Although Knowledge Management was judged relatively low in importance, its presence on the performance map reminds everyone that they need to capture the results of the project as part of the accumulating knowledge capital of the firm.The Concept System process enabled this group in less than four hours of participant time to develop a comprehensive map of the performance domain for this project. The map tells the "performance story" that the group expects. It shows which performance goal areas the stakeholders believe are relatively more important, and where they disagree on importance. It provides a comprehensive framework for the development of specific performance measures for the project and, ultimately, for assessing the business and financial returns the project generates. The Concept System is not the only way to map the performance domain, but we believe it is the best approach for accomplishing this efficiently and rigorously.
2. Develop performance hierarchy.
The performance hierarchy describes the intervention’s performance goals, objectives and measures. Why is a hierarchy needed? If we just knew the overall project costs and overall financial benefits, we could easily calculate the overall financial returns – assuming we obtained perfect performance. But we expect that the performance from an intervention will seldom be perfect and, further, that performance from one goal or objective to another will differ even under the best circumstances. While we can estimate financial returns in the aggregate (e.g., for the whole project), performance is almost always measured at a much lower level of aggregation (i.e., for specific performance objectives). A performance hierarchy enables us to aggregate specific measures of human performance across objectives, goals and for the entire project. With this information, we are able to distribute costs and benefits throughout the hierarchy and weight the benefits by the level of performance.
If The Concept System was used to map the performance domain, we already have the performance hierarchy developed – it consists of the Performance Goal Areas (i.e., map clusters) and Performance Objectives (i.e., map statements) as described above. Each objective is automatically contained within a goal area on the map. We can see how this translates into the software tools in the screenshot The Concept System Performerã program for performance management and measurement shown in Figure 7. The top panel shows the Performance Goal Areas and Performance Objectives from the maps shown above. We see that the currently selected objective Have strong project leadership within the goal area GHPD Program Team has two performance measures defined for it (as shown in the bottom panel). It is not necessary to define performance measures for every objective, although in some situations that may be desirable. Depending on the needs of the project we might define one or more measures for only selected objectives. Usually we will want to define at least one measure within each goal area.
The performance hierarchy is the major mechanism for linking specific performance measures at the individual or department level with project-level estimates of costs and benefits. The process for accomplishing this linkage is described in subsequent steps.
Figure 7. Screen showing the performance hierarchy from the concept mapping project.
3. Weight performance objectives.
In this step, we want to weight each performance objective for its contribution to the overall financial return of the project. This is inevitably a subjective rating, but we can increase its rigor by doing a thorough consensus analysis as shown above in the pattern matching example. In this example, the overall stakeholder importance rating is used as the estimate of the relative contribution of each objective and goal area to the financial returns of the project. In order to distribute this weight proportionately, we first convert the rating metric by normalizing the average ratings. The formula for normalizing the weights is:
Weight = Ave/Sum
where the Ave is the average importance for a specific objective. As a result of this normalization, each weight will be the proportion (between 0 and 1) that the statement contributes to the financial return. This weighting is handled automatically in The Concept System Performer software.
4. Calculate total costs.
Total costs for the project are calculated using standard accounting methods. In the GHPD example, we would simply total the expenditures from the appropriate accounts for the project. In most cases, calculation of total costs will be a simple accounting matter. However, when looking at financial returns over multiple accounting cycles (e.g., for several years), the analyst will have to determine how costs will be distributed throughout the entire life of the project. Because project costs will tend to be more heavily expended in the initial periods of the projects (and returns might be expected considerably later), it may be desirable to amortize costs over the life of the project.
5. Estimate total benefit.
In this step, we want to estimate the overall financial benefit expected from the project. The best way to accomplish this is to involve a group of client stakeholders who are familiar with the project in an iterative Delphi method estimation process. The process is very simple and takes little time. The initial instruction for the first round of estimation for the GHPD project example might be:
Imagine that this project works exactly as we hope it will. Think about what the effects of this project over the next 12 months, assuming that we get perfect performance (i.e., the project works perfectly). I would like you to discuss the different ways this project will affect the financial performance of the company over the next 12 months under these ideal circumstances. After you’ve discussed the potential financial effects, I would like each one of you to estimate a single dollar value that represents your best guess of the maximum financial benefit of this project over the entire 12 month period. Don’t worry about this initial estimate – you will have a chance to revise your estimate in the next round.
Figure 8. First round of Delphi procedure to estimate total benefits.
Figure 8 shows how the data for the first round are entered into the Performer program. The benefit estimate for each person can be entered. The estimated total cost is obtained either from financial predictions or budgets, or is obtained from accounting records in post hoc studies.
The results for the first round estimates are shown in Figure 9.
Figure 9. Results of first Delphi round of benefits estimates.
We can see that there were nine participants in this round. The figure shows that the average benefit for this round is $143,444.44 and indicates that the total estimated cost is $50,000. Notice that the program also gives the range of values (i.e., the 95% confidence interval) around the average that constitute the reasonable lower and upper limits for the estimate.
The B/C Ratio and ROI are also shown. These values represent the financial benefit of the project assuming a perfect result (i.e., 100% performance). These estimates constitute the upper limits of what we might reasonably expect for this project, given the Round 1 benefit estimate. Again, the range of reasonable values for both ratios is also given.
Since this is only the first round of estimates, it will always be desirable to go to the next round. A stopping rule is used to determine when enough estimation rounds have been accomplished. Figure 10 shows the benefits estimation results after the fourth round of estimates.
Figure 10. Results of first Delphi round of benefits estimates.
This time the average benefit estimate is $140,000. The stopping rule indicates that results did not differ significantly from the previous round. Consequently, the Delphi procedure can stop at this fourth round estimate.
This estimate of benefits is obviously only as good as the judgment of the estimators. That’s why it is critical that the participants be familiar with both the program and with the finances of the organization. If cost accounting procedures are used and the dollar benefits of the program can be calculated directly, you can simply enter the actual value instead of following this estimation procedure. However, as discussed above, it is unlikely that cost accounting approaches can be used in most circumstances. Even if they are, because of the judgmental nature of some of the cost accounting procedures, it may still be desirable to enter a range of cost accounting benefit estimates each of which is based on different assumptions.
6. Distribute costs and benefits across hierarchy.
In step 5 we estimated financial results under ideal circumstances. That is, we assumed that performance was perfect for the program. However, this is unlikely to be the case in most situations. In this step, we begin to take performance across the performance hierarchy into account.
In order to distribute the costs and benefits across the various performance objectives (and goal areas), we have to determine how much each performance objective is likely to impact financial results. We could engage a group of stakeholders to estimate this impact directly, or we could use another estimator as a proxy for a direct estimate. For instance, in this example, we use the original importance rating for each objective as a proxy for the financial impact of the objective. The assumption here is that an objective that is more important will generally have a greater impact on financial performance (and that’s why it’s judged more important).
For each objective, we take the average importance rating of the entire stakeholder group as the proxy for its weight on financial impact. We next want to distribute the total costs and total benefits across all of the performance objectives. You can think of the total cost or total benefit as a pie. Essentially, we want to estimate how much of the pie is influenced by each objective.
We can estimate the proportion for each objective by normalizing the average ratings of importance. The formula for the weight is the normalized score where:
Weight = Average/(Sum of Averages)
Let’s look at a simple example. Taking the fourth round benefit estimate of $140,000 and the cost estimate of $50,000, consider a single objective that has an average importance rating of 3.92 on a 1-to-5 scale). We calculate the normalized weight and obtain a score of .0516. This means that this specific objective is estimated to contribute about 5.16% of the total costs or benefits. Given the total values, we would estimate that the total costs attributable to this objective are .0516 * $50,000 = $2,578.80 and the total benefits are .0516 * $140,000 = $7,220.63.
Once we obtain the weights for the objectives, we can easily determine the weights for each goal area by adding its objective’s weights.
There are a few caveats to be aware of with this proportioning approach. First, the weights are only as good as the variable on which they are based. If the average importance rating is not a reasonable estimate of the financial contribution of each objective, another variable should be used. Second, there are likely to be differences of opinion about the relative contributions associated with each objective (just as there are differences of opinion about relative importance). To some extent, this is mitigated by the fact that we are estimating ranges for financial results rather than a single estimate. Nevertheless, it might make sense to conduct the analysis using several different estimates of the weights to see how much they affect final results. Finally, it is important to recognize that this procedure estimates the total benefit contribution of each objective assuming a perfect result – that performance is 100%. In step 8 below, we estimate benefits for the performance that we actually measure.
The Performer program automatically computes weights for a given weighting variable.
7. Measure performance on standardized scale.
When the performance improvement intervention is implemented, performance measures are gathered, usually at regular time intervals (e.g., weekly, monthly, quarterly). Each performance measure is linked to a performance objective in the hierarchy and, through that objective, to a performance goal. In any project each objective may have no measure, one measure, or multiple measures attached to it.
Performance measures are usually collected on a variety of scales. Some measures may be subjective, as in a 1-to-5 rating of employee performance. Other measures are more objective. For instance, if the goal area is Safety and the objective is to reduce the number of accidents on the factory floor each month, the measure might be the count of the number of specific types of accidents that occur each month.
Since performance is measured on a variety of scales, we need to transform each measure to a standardized scale in order to aggregate and compare across measures. Usually we convert all measures to a percentage of ideal performance scale where 0 equals the worst level of performance, 100% equals ideal performance, and other values indicate the percent of ideal performance.
As each wave of performance measures is collected, we can automatically estimate the financial results to date. It is important to note that if we estimated benefits and cost for a 12 month period but measure on a monthly basis, the financial results would be estimates of what we would expect at the end of 12 months given the currently measured levels of performance. Ideally we would hope that performance will improve over time and, as it does, we will observe financial results improving. If it is desirable to estimate intermediate financial results (e.g., at the end of each month), it will be necessary to use estimates of benefit and cost for the desired time period.
8. Estimate financial returns across hierarchy.
Once we have measures of performance we are able to obtain estimates of actual financial returns of the project. For each performance objective that has an associated performance measure, we first average the performance scores (obviously, if the objective has a single measure, that measure’s performance score will be used).
From step 6 above, we already have distributed total costs and benefits across all objectives that have measures. In this step, we weight the benefits for that objective by the performance for that objective. Since performance is measured on a standardized scale of 0-100, if we divide the performance score by 100 we can obtain a performance weight for the objective. For instance, if the performance for an objective is currently at 60%, the weight would be .6.
To obtain estimates of the financial results for an objective, we would multiply the dollar benefits associated with that measure by the performance weight. Note that we would not multiply costs. We assume that costs are fixed and are not affected by different performance levels. Let’s see how this works for the example given in Step 6 above. There, we used the fourth round total benefit estimate of $140,000 and a total cost estimate of $50,000. Our sample objective had a weight of .0515, or slightly more than 5% of the total and so the estimate of total costs attributable to this objective are .0516 * $50,000 = $2,578.80 and the total benefits are .0516 * $140,000 = $7,220.63.
Now, let’s weight the benefits for the performance actually observed. Assume that for the period in question (i.e., the first 12 months), we obtain 60% performance on this objective. The weighted benefit would then be .6 * $7,220.63 = $4,332.38. Now, we are able to compute the financial results for this objective. The B/C ratio will be the performance-weighted benefit divided by the cost, or, $4,332.38/$2,578.80 = 1.68. This means that for every dollar spent addressing the objective, we expect to earn $1.68 dollars back. The ROI ratio is simply the (Benefit-Cost)/Cost which in this case is ($4,332.38 - $2,578.80)/ $2,578.80 = .68. This means that for every dollar spent addressing that objective, we estimate that we earn 68 cents in profit.
Because we have constructed the entire performance hierarchy, we are able to "roll the estimates up" to the performance goal areas and, thereby, for the entire project. That is, we are able to estimate financial results for the entire project taking into account the obtained performance across all of the measures, objectives, and goal areas.
Conclusions
While the detailed description of the eight steps suggests that this procedure for estimating financial returns is a complicated one, it is actually quite simple to implement in practice, assuming you have taken the time to develop a performance hierarchy. Once a hierarchy exists, all that’s needed is an estimate of total costs and benefits for the project. Total costs should be relatively easy to obtain. Before implementation, one could use the budgeted amount for the program as an estimate. After the program is implemented, one simply uses the accounted costs for the project. To estimate benefits requires the Delphi procedure described earlier. This is a relatively simple process that should be easy to accomplish in less than an hour of participant time.
The "bottom line" here is that a good performance measurement system will enable relatively easy estimation of financial results – there is little additional marginal cost to estimating financial outcomes, assuming you have a well-constructed measurement system. The Concept System approach is designed so that the performance hierarchy is correctly constructed. Adding in the estimation of financial returns is then a relatively simple and inexpensive addition that yields critical information about the financial impacts of the performance improvement project.
