| related case studies |
This project was an evaluation of a course in computer system design. Nine persons familiar with the content generated 66 statements describing the different topics that should be included in the training. The map divided these statements into seven clusters. Four of the clusters pertained directly to the design of the system itself and were rated high in importance. The other three clusters related to understanding the approach generally or to managing the engagement.
The course was evaluated in three pilot implementations (alpha, beta and pilot 1) and two actual conducts (Conducts 2 and 3). The results indicated convincingly that the program was having an effect. The pattern match of importance ratings versus achievement for Conduct 3 shows a strong correspondence between what was rated important and what showed the strongest achievement levels.
Across the five separate implementations, the outcome pattern match remained consistently strong, varying from .74 to .90 at the highest.
Along with that, it is apparent that the measurement of achievement is fairly reliable with consistency pattern matches ranging from .89 to .96. There is evidence that there was a slight decline in scores in the pilot and in Conduct 3, even though the order of outcomes holds across variables. This could be due to a number of factors -- tougher raters at those times, different instructors or class circumstances. The evaluation team will continue to monitor these rates to see if they represent a trend or an anomaly.
Probably the more critical time series to monitor is the outcome pattern matching correlation. If it falls below .65, the evaluation and development teams should examine carefully which clusters are disconnecting. If it falls suddenly below .6, the program should be stopped temporarily until a clear determination can be made regarding what is going wrong.
The strong and consistent outcome pattern match (tracked over time) indicates that the training is having an effect and that the effect is consistent.