related case studies

Training planning and evaluation Systems design course Training in use of a knowledge base

This project evaluated an introductory-level training program for consultants who do systems analysis for computer information systems. A group of 23 stakeholders who were familiar with the intended course content generated 56 statements and developed a map that divided the course content into nine clusters. The Client Issues cluster was rated most important. In general, technical topics were on the top of the map and team/individual topics were on the bottom.

Figure 1. Concept map for an Introductory Computer Systems Design Course.

Two stakeholder groups were asked to rate the importance of the course topics -- the faculty (who were chosen to teach the course because they were experienced consultants) and the developers (who have varying experience in the subject matter). The consensus pattern match shows only a slight to moderate degree of agreement between these groups. In general, the developers prioritized more on personal and technical topics while the faculty emphasized the importance of client and team issues.

Figure 2. Consensus pattern match between faculty and developers for an Introductory Computer Systems Analysis Course.

Participants in an early implementation of the course rated both their opportunity and achievement with respect to the topics in the course. The consistency across these two outcome ratings was extremely high (.987) indicating good reliability.

Figure 3. Consistency pattern match between opportunity and achievement outcome ratings for an Introductory Computer Systems Analysis Course.

Two outcome pattern matches were computed, one for opportunity and one for achievement. Both indicated a moderate to strong training program effect. And both indicated a similar pattern of outcomes (not too surprising given the near-perfect match between the two outcomes).

Figure 4. Opportunity outcome pattern match for an Introductory Computer Systems Analysis course.

Figure 5. Achievement outcome pattern match for an Introductory Computer Systems Analysis course.

Despite the good evidence for the effect of the training, it can be improved somewhat. Three key clusters are limiting the match considerably -- Personal Awareness and Skill, Client Issues, and Client Server Issues. All three can be emphasized slightly more in the course, relative to the other topics, in order to improve on the effect of the training beyond what it currently evidences.