The joint Committee of Programmed Instruction
and Teaching Machines in 1963 recommended distinguishing between internal
characteristics of the PI program (which can be judged by an inspection
of the program) and external characteristics (features which are only
revealed by the performance of users after the completion of the program,
or by objective features). Many of the same selection criteria can
be used when evaluating CAI programs or designing web-based courses.
Internal
Is the subject matter appropriate for,
or needed by, the intended audience? Are the objectives of the program,
if stated, appropriate for or needed by the intended audience? Are
they relevant to some real-world need or prerequisite to further study?
Is the sequence of frames consistent with the structure of the subject-matter
content? Is the sequence of frames appropriate for the type of users
in the intended audience? Is the step size appropriate for the ability
of the users?
Are the response opportunities frequent
enough for the type of users or the structure of the content? When
a response is required, does feedback follow at some appropriate point,
not too delayed from the response? Are there any provisions for individualizing
the sequence or content of the program (e.g., branching, diagnostic
tests, surveys) or is every user required to proceed through each
step? Is an instructor's guide available that contains recommendations
for administration, placement, or integration with other study materials?
Is a posttest available which measures attainment of the objectives?
External
Is there evidence of any testing or
user tryout of the program prior to distribution? Was the group used
to field-test the program similar to the group for which you are selecting
a program, in terms of age, ability, computer experience, and prior
knowledge? Did the program, as tested in the tryout, produce a sufficient
level of learning? The National Education Association cautioned that
an internal inspection alone can be deceptive. External data from
tests involving users at your institution or a similar group should
be included in the evaluation process.*20 The users should represent,
as closely as possible, the intended population of users.
Other
Criteria
Are the directions on how to use the
program clear, adequate, and appropriate for the target users? Are
the technical terms and symbols adequately defined? Does the content
cover the stated objectives without unnecessary redundancy and superfluous
information? (Some redundancy for the purpose of practice is a normal
part of PI.) Does the program periodically review what has been taught
most recently throughout the program? Does the achievement test measure
the user's comprehension of the content of the program as opposed
to mere recall of facts? Does the program allow for individual differences
in learning other than pacing, such as prior knowledge or background,
ability level, and learning styles? Are active responses required
of the user? Do they require comprehension of the material, rather
than mere recall or copying of information?
Are the responses an appropriate form
for the users and relevant to the objectives? Are there enough responses
for each skill or unit of information? Do the questions require the
user to utilize the critical information in the frame, or to recall
only trivial facts? Are the prompts necessary for making the correct
response, or do they replace the thought processes in which you want
users to engage? Are there too many prompts? Are enough questions
asked to allow the user to apply the information to a variety of situations?
Will the skills taught in the program transfer to a real-world setting?
Is there at least one question requiring users to respond to each
important piece of information or skill? Is the feedback accurate
and clear to the user? It is surprising how often this simple criterion
is violated. Does it immediately follow the response? Can the feedback
be seen by the user while responding? Are alternative answers to questions
acceptable and are the acceptable responses indicated? Are helpful,
remedial comments provided when wrong or alternative answers are anticipated?
Do the instructions allow the user to skip unnecessary repetition
or to return to material when necessary? If a nonprint medium is used,
is it an appropriate format for the content of the program? Could
the program be more effectively delivered in another format?
Is the delivery mode (machine/text,
CAI, web-based) or control of pacing (self-paced, programmed, or group-paced)
appropriate for the maturity level and ability of the intended users?
Are there a sufficient number of review questions and feedback to
keep all parts of the program current and active in memory?
