03162nas a2200181 4500008004100000245010600041210006900147260005500216520249300271653000802764653001502772653002702787100002202814700002602836700001602862700001502878856008702893 2017 eng d00aEfficiency of Targeted Multistage Calibration Designs under Practical Constraints: A Simulation Study0 aEfficiency of Targeted Multistage Calibration Designs under Prac aNiigata, JapanbNiigata Seiryo Universityc08/20173 a

Calibration of an item bank for computer adaptive testing requires substantial resources. In this study, we focused on two related research questions. First, we investigated whether the efficiency of item calibration under the Rasch model could be enhanced by calibration designs that optimize the match between item difficulty and student ability (Berger, 1991). Therefore, we introduced targeted multistage calibration designs, a design type that refers to a combination of traditional targeted calibration designs and multistage designs. As such, targeted multistage calibration designs consider ability-related background variables (e.g., grade in school), as well as performance (i.e., outcome of a preceding test stage) for assigning students to suitable items.

Second, we explored how limited a priori knowledge about item difficulty affects the efficiency of both targeted calibration designs and targeted multistage calibration designs. When arranging items within a given calibration design, test developers need to know the item difficulties to locate items optimally within the design. However, usually, no empirical information about item difficulty is available before item calibration. Owing to missing empirical data, test developers might fail to assign all items to the most suitable location within a calibration design.

Both research questions were addressed in a simulation study in which we varied the calibration design, as well as the accuracy of item distribution across the different booklets or modules within each design (i.e., number of misplaced items). The results indicated that targeted multistage calibration designs were more efficient than ordinary targeted designs under optimal conditions. Especially, targeted multistage calibration designs provided more accurate estimates for very easy and 52 IACAT 2017 ABSTRACTS BOOKLET very difficult items. Limited knowledge about item difficulty during test construction impaired the efficiency of all designs. The loss of efficiency was considerably large for one of the two investigated targeted multistage calibration designs, whereas targeted designs were more robust.

References

Berger, M. P. F. (1991). On the efficiency of IRT models when applied to different sampling designs. Applied Psychological Measurement, 15(3), 293–306. doi:10.1177/014662169101500310

Session Video

10aCAT10aEfficiency10aMultistage Calibration1 aBerger, Stephanie1 aVerschoor, Angela, J.1 aEggen, Theo1 aMoser, Urs uhttps://drive.google.com/file/d/1ko2LuiARKqsjL_6aupO4Pj9zgk6p_xhd/view?usp=sharing01251nas a2200181 4500008004100000245012100041210006900162260001200231520055600243653003300799653000800832653001900840653003500859100001800894700001600912700002200928856011900950 2011 eng d00aItem Selection Methods based on Multiple Objective Approaches for Classification of Respondents into Multiple Levels0 aItem Selection Methods based on Multiple Objective Approaches fo c10/20113 a

Is it possible to develop new item selection methods which take advantage of the fact that we want to classify into multiple categories? New methods: Taking multiple points on the ability scale into account; Based on multiple objective approaches.

Conclusions

10aadaptive classification test10aCAT10aitem selection10asequential classification test1 aGroen, Maaike1 aEggen, Theo1 aVeldkamp, Bernard uhttp://www.iacat.org/content/item-selection-methods-based-multiple-objective-approaches-classification-respondents00781nas a2200205 4500008004100000245003400041210003200075260001200107520027300119653000800392653000800400653002300408653001000431653001200441653000800453100002200461700001900483700001600502856005700518 2011 eng d00aA Test Assembly Model for MST0 aTest Assembly Model for MST c10/20113 a

This study is just a short exploration in the matter of optimization of a MST. It is extremely hard or maybe impossible to chart influence of item pool and test specifications on optimization process. Simulations are very helpful in finding an acceptable MST.

10aCAT10amst10amultistage testing10aRasch10arouting10atif1 aVerschoor, Angela1 aRadtke, Ingrid1 aEggen, Theo uhttp://www.iacat.org/content/test-assembly-model-mst01621nas a2200217 4500008004100000020002200041245008200063210006900145260002600214300001200240490000700252520088700259653002801146653002501174653002501199653001901224653001601243100001601259700002501275856010301300 2006 eng d a0146-6216 (Print)00aOptimal testing with easy or difficult items in computerized adaptive testing0 aOptimal testing with easy or difficult items in computerized ada bSage Publications: US a379-3930 v303 aComputerized adaptive tests (CATs) are individualized tests that, from a measurement point of view, are optimal for each individual, possibly under some practical conditions. In the present study, it is shown that maximum information item selection in CATs using an item bank that is calibrated with the one- or the two-parameter logistic model results in each individual answering about 50% of the items correctly. Two item selection procedures giving easier (or more difficult) tests for students are presented and evaluated. Item selection on probability points of items yields good results only with the one-parameter logistic model and not with the two-parameter logistic model. An alternative selection procedure, based on maximum information at a shifted ability level, gives satisfactory results with both models. (PsycINFO Database Record (c) 2007 APA, all rights reserved)10acomputer adaptive tests10aindividualized tests10aItem Response Theory10aitem selection10aMeasurement1 aEggen, Theo1 aVerschoor, Angela, J uhttp://www.iacat.org/content/optimal-testing-easy-or-difficult-items-computerized-adaptive-testing01310nas a2200169 4500008004100000245009500041210006900136300001200205490000700217520070700224653003400931100001400965700001600979700001600995700001701011856011201028 2002 eng d00aEvaluation of selection procedures for computerized adaptive testing with polytomous items0 aEvaluation of selection procedures for computerized adaptive tes a393-4110 v263 aIn the present study, a procedure that has been used to select dichotomous items in computerized adaptive testing was applied to polytomous items. This procedure was designed to select the item with maximum weighted information. In a simulation study, the item information function was integrated over a fixed interval of ability values and the item with the maximum area was selected. This maximum interval information item selection procedure was compared to a maximum point information item selection procedure. Substantial differences between the two item selection procedures were not found when computerized adaptive tests were evaluated on bias and the root mean square of the ability estimate. 10acomputerized adaptive testing1 aRijn, P W1 aEggen, Theo1 aHemker, B T1 aSanders, P F uhttp://www.iacat.org/content/evaluation-selection-procedures-computerized-adaptive-testing-polytomous-items01598nas a2200157 4500008004100000245008200041210006900123300001100192490000700203520101400210653003401224653004001258100001601298700002401314856010201338 2000 eng d00aComputerized adaptive testing for classifying examinees into three categories0 aComputerized adaptive testing for classifying examinees into thr a713-340 v603 aThe objective of this study was to explore the possibilities for using computerized adaptive testing in situations in which examinees are to be classified into one of three categories.Testing algorithms with two different statistical computation procedures are described and evaluated. The first computation procedure is based on statistical testing and the other on statistical estimation. Item selection methods based on maximum information (MI) considering content and exposure control are considered. The measurement quality of the proposed testing algorithms is reported. The results of the study are that a reduction of at least 22% in the mean number of items can be expected in a computerized adaptive test (CAT) compared to an existing paper-and-pencil placement test. Furthermore, statistical testing is a promising alternative to statistical estimation. Finally, it is concluded that imposing constraints on the MI selection strategy does not negatively affect the quality of the testing algorithms10acomputerized adaptive testing10aComputerized classification testing1 aEggen, Theo1 aStraetmans, G J J M uhttp://www.iacat.org/content/computerized-adaptive-testing-classifying-examinees-three-categories