related case studies

Systems requirements course Training planning & evaluation

The purpose of this study was to evaluate the effectiveness of a self-study training program in computer programming. Nine persons familiar with the subject matter generated 71 statements describing different programming concepts that needed to be learned. The resulting nine cluster map includes general programming issues on the left (use of resources, personal work style, coding basics, testing and debugging) and more technical concepts related to the computer language on the right. The clusters related to the basics of the language itself and the basics of programming in general were rated highest in importance.

Figure 1. Concept map for a self-study course in computer programming.

The results indicated that the training was not achieving what was expected. In four separate conducts of the training there was virtually no match between what was expected and what participants reported they achieved (outcome pattern matching r’s = -.079, -.046, -.059, and .128 respectively). For instance, although Coding Basics and Functions and Modularization were two of the three most important clusters, they placed fifth and eighth (out of ten) in achievement. Conversely, although Personal Work Style and SQL were rated fifth and tenth in importance, they placed second and third in achievement.

Figure 2. Pattern match of importance and achievement for conduct 1 of the self-study programming course.

And, it’s not easy to dismiss these results as due to chance or poor measurement. The achievement ratings across the four conducts were amazingly consistent, with all cross-conduct pattern matches exceeding .92.

The problem, however, may not be as much with the training as with the initial conceptualization of it. The development team is currently looking at what was initially thought to be important to see if those ratings still make sense. In addition, it may be that the specific audience for the training is more experienced than originally expected, especially in other languages than the one for this training. If so, it might help explain why participants didn’t report much achievement in the key areas of Coding Basics and Functions and Modularization -- they already knew much of that coming into the course. Current efforts are underway to better understand whether the lack of expected outcomes is due to some misspecification in their estimation or to inadequate measurement of achievement for this population.

Figure 3. Consistency pattern match of achievement ratings across four conducts of the self-study computer programming course.