Computerized Adaptive Testing: A Primer

Rating:
Summary: format needs work
Review: Great information. But the format needs work. Author credentials and degrees should be included. Author email also. Very lousy format for graphs, which do have useful information. However, the print and keys are literally fuzzy. Perhaps the paper absorbed too much ink, black and white is not sufficient, and the symbols for different concepts are too much alike. So far I have not found mention of NCLEX or applications in nursing.

Rating:
Summary: Good summary of CAT and IRT
Review: If you want to learn more about computer adaptive testing (CAT), this book is the place to start. Chapter four, the best chapter in the book, provides an excellent introduction to item response theory (IRT). If the chapter on IRT seems hard to understand and full of strange mathematical symbols, this reflects the unfortunate fact that ALL the literature on IRT has these shortcomings. This book could contain a lot more information. Outside of chapter four, the information is only marginally useful. Nevertheless, it's the best book on CAT that's currently available. I recommend it for anyone who wants to learn about CAT and IRT. However, if you are weak in math and statistics, you probably won't understand most of the book.

Rating:
Summary: Good, practical guide on the theory of CAT
Review: This book contains most of what you need to understand how computerized adaptive tests (CAT) do what they do. It is a good complement to the seminal book on Item Response Theory (IRT) by Wainer's colleague, Frederic Lord (Applications of Item Response Theory to Practical Testing Problems).

The assumptions that go into item characterizations and test designs are discussed here; these assumptions are frequently glossed over or accepted as "fact" by many in the psychometric field. My own interests are in the type of test called a Mastery test. The test design criteria for a Mastery test are different than those for a general achievement test (e.g., the SAT). The stopping rules, the range of item difficulties, and the next-item selection algorithm should be tailored to the needs of a Mastery test, but this is not always the case. One consequence is that if an examinee misses several items early on in a poorly designed Mastery test, it may be difficult or impossible to recover, because the items at the beginning usually carry the most "weight" toward the final score or grade. Wainer gives a good description of the design criteria for different situations and how to avoid various problems and pitfalls.

Chapter 6, Scaling and Equating, shows the reader how an open-ended logarithmic scoring scale - used by any CAT based on IRT - is converted to a fixed scale such as 0-100 or 0-1000. A variety of methods are available, each with its own advantages and disadvantages.

This primer is cited by the Microsoft Corporation in their electronic white paper on adaptive testing (along with Lord's book and several other references.) Microsoft also has a sample adaptive test available. This test has at least three incorrect answers in it, and the impact of giving the real correct answer to an item, but having it graded as incorrect, is easily seen. Other items on the Microsoft sample CAT have assigned item difficulties that are out of line with their true difficulties. These types of problems are covered in Wainer's book but not in any of the other references on adaptive testing or IRT that I have perused.

A nice touch in the primer is a chapter-by-chapter "running story" about Cindy and Scott who apply for jobs and go through the adaptive-test-taking experience. The story line is constructed to fit in with the concepts covered in each chapter. Another feature in the book is the inclusion of Exercise/Study Questions at the end of each chapter. Many books on IRT include similar questions and exercises, probably a consequence of being written by educators who write textbooks. Wainer does not include the answers, but they can be deduced if one actually reads the book.

The book does not provide computer algorithms or pseudocode that would allow one to actually create an adaptive test Hopefully, a future edition will provide better guidance in this area. Although almost a decade since its initial publication, Wainer's book is still a good reference book for anyone interested in computer adaptive testing.