01382nas a2200133 4500008003900000245010400039210006900143300001200212490000700224520093300231100001701164700002201181856004501203 2020 d00aThe Optimal Item Pool Design in Multistage Computerized Adaptive Tests With the p-Optimality Method0 aOptimal Item Pool Design in Multistage Computerized Adaptive Tes a955-9740 v803 aThe present study extended the p-optimality method to the multistage computerized adaptive test (MST) context in developing optimal item pools to support different MST panel designs under different test configurations. Using the Rasch model, simulated optimal item pools were generated with and without practical constraints of exposure control. A total number of 72 simulated optimal item pools were generated and evaluated by an overall sample and conditional sample using various statistical measures. Results showed that the optimal item pools built with the p-optimality method provide sufficient measurement accuracy under all simulated MST panel designs. Exposure control affected the item pool size, but not the item distributions and item pool characteristics. This study demonstrated that the p-optimality method can adapt to MST item pool design, facilitate the MST assembly process, and improve its scoring accuracy.1 aYang, Lihong1 aReckase, Mark, D. uhttps://doi.org/10.1177/001316441990129202064nas a2200157 4500008003900000245009100039210006900130300001200199490000700211520157600218100001501794700001901809700001401828700001901842856004501861 2020 d00aStratified Item Selection Methods in Cognitive Diagnosis Computerized Adaptive Testing0 aStratified Item Selection Methods in Cognitive Diagnosis Compute a346-3610 v443 aCognitive diagnostic computerized adaptive testing (CD-CAT) aims to obtain more useful diagnostic information by taking advantages of computerized adaptive testing (CAT). Cognitive diagnosis models (CDMs) have been developed to classify examinees into the correct proficiency classes so as to get more efficient remediation, whereas CAT tailors optimal items to the examinee’s mastery profile. The item selection method is the key factor of the CD-CAT procedure. In recent years, a large number of parametric/nonparametric item selection methods have been proposed. In this article, the authors proposed a series of stratified item selection methods in CD-CAT, which are combined with posterior-weighted Kullback–Leibler (PWKL), nonparametric item selection (NPS), and weighted nonparametric item selection (WNPS) methods, and named S-PWKL, S-NPS, and S-WNPS, respectively. Two different types of stratification indices were used: original versus novel. The performances of the proposed item selection methods were evaluated via simulation studies and compared with the PWKL, NPS, and WNPS methods without stratification. Manipulated conditions included calibration sample size, item quality, number of attributes, number of strata, and data generation models. Results indicated that the S-WNPS and S-NPS methods performed similarly, and both outperformed the S-PWKL method. And item selection methods with novel stratification indices performed slightly better than the ones with original stratification indices, and those without stratification performed the worst.1 aYang, Jing1 aChang, Hua-Hua1 aTao, Jian1 aShi, Ningzhong uhttps://doi.org/10.1177/014662161989378301943nas a2200145 4500008003900000245007700039210006900116300001200185490000700197520148100204100002101685700002301706700002301729856004501752 2019 d00aComputerized Adaptive Testing for Cognitively Based Multiple-Choice Data0 aComputerized Adaptive Testing for Cognitively Based MultipleChoi a388-4010 v433 aCognitive diagnosis models (CDMs) are latent class models that hold great promise for providing diagnostic information about student knowledge profiles. The increasing use of computers in classrooms enhances the advantages of CDMs for more efficient diagnostic testing by using adaptive algorithms, referred to as cognitive diagnosis computerized adaptive testing (CD-CAT). When multiple-choice items are involved, CD-CAT can be further improved by using polytomous scoring (i.e., considering the specific options students choose), instead of dichotomous scoring (i.e., marking answers as either right or wrong). In this study, the authors propose and evaluate the performance of the Jensen–Shannon divergence (JSD) index as an item selection method for the multiple-choice deterministic inputs, noisy “and” gate (MC-DINA) model. Attribute classification accuracy and item usage are evaluated under different conditions of item quality and test termination rule. The proposed approach is compared with the random selection method and an approximate approach based on dichotomized responses. The results show that under the MC-DINA model, JSD improves the attribute classification accuracy significantly by considering the information from distractors, even with a very short test length. This result has important implications in practical classroom settings as it can allow for dramatically reduced testing times, thus resulting in more targeted learning opportunities.1 aYigit, Hulya, D.1 aSorrel, Miguel, A.1 ade la Torre, Jimmy uhttps://doi.org/10.1177/014662161879866501532nas a2200169 4500008003900000245007100039210006900110300001200179490000700191520103000198100001601228700001801244700001801262700001801280700001901298856004501317 2018 d00aLatent Class Analysis of Recurrent Events in Problem-Solving Items0 aLatent Class Analysis of Recurrent Events in ProblemSolving Item a478-4980 v423 aComputer-based assessment of complex problem-solving abilities is becoming more and more popular. In such an assessment, the entire problem-solving process of an examinee is recorded, providing detailed information about the individual, such as behavioral patterns, speed, and learning trajectory. The problem-solving processes are recorded in a computer log file which is a time-stamped documentation of events related to task completion. As opposed to cross-sectional response data from traditional tests, process data in log files are massive and irregularly structured, calling for effective exploratory data analysis methods. Motivated by a specific complex problem-solving item “Climate Control” in the 2012 Programme for International Student Assessment, the authors propose a latent class analysis approach to analyzing the events occurred in the problem-solving processes. The exploratory latent class analysis yields meaningful latent classes. Simulation studies are conducted to evaluate the proposed approach.1 aXu, Haochen1 aFang, Guanhua1 aChen, Yunxiao1 aLiu, Jingchen1 aYing, Zhiliang uhttps://doi.org/10.1177/014662161774832506631nas a2200157 4500008004100000245011400041210006900155260005500224520601700279653001106296653004106307653001906348100001706367700001806384856007106402 2017 eng d00aEfficiency of Item Selection in CD-CAT Based on Conjunctive Bayesian Network Modeling Hierarchical attributes0 aEfficiency of Item Selection in CDCAT Based on Conjunctive Bayes aNiigata, JapanbNiigata Seiryo Universityc08/20173 a

Cognitive diagnosis models (CDM) aim to diagnosis examinee’s mastery status of multiple fine-grained skills. As new development for cognitive diagnosis methods emerges, much attention is given to cognitive diagnostic computerized adaptive testing (CD-CAT) as well. The topics such as item selection methods, item exposure control strategies, and online calibration methods, which have been wellstudied for traditional item response theory (IRT) based CAT, are also investigated in the context of CD-CAT (e.g., Xu, Chang, & Douglas, 2003; Wang, Chang, & Huebner, 2011; Chen et al., 2012).

In CDM framework, some researchers suggest to model structural relationship between cognitive skills, or namely, attributes. Especially, attributes can be hierarchical, such that some attributes must be acquired before the subsequent ones are mastered. For example, in mathematics, addition must be mastered before multiplication, which gives a hierarchy model for addition skill and multiplication skill. Recently, new CDMs considering attribute hierarchies have been suggested including the Attribute Hierarchy Method (AHM; Leighton, Gierl, & Hunka, 2004) and the Hierarchical Diagnostic Classification Models (HDCM; Templin & Bradshaw, 2014).

Bayesian Networks (BN), the probabilistic graphical models representing the relationship of a set of random variables using a directed acyclic graph with conditional probability distributions, also provide an efficient framework for modeling the relationship between attributes (Culbertson, 2016). Among various BNs, conjunctive Bayesian network (CBN; Beerenwinkel, Eriksson, & Sturmfels, 2007) is a special kind of BN, which assumes partial ordering between occurrences of events and conjunctive constraints between them.

In this study, we propose using CBN for modeling attribute hierarchies and discuss the advantage of CBN for CDM. We then explore the impact of the CBN modeling on the efficiency of item selection methods for CD-CAT when the attributes are truly hierarchical. To this end, two simulation studies, one for fixed-length CAT and another for variable-length CAT, are conducted. For each studies, two attribute hierarchy structures with 5 and 8 attributes are assumed. Among the various item selection methods developed for CD-CAT, six algorithms are considered: posterior-weighted Kullback-Leibler index (PWKL; Cheng, 2009), the modified PWKL index (MPWKL; Kaplan, de la Torre, Barrada, 2015), Shannon entropy (SHE; Tatsuoka, 2002), mutual information (MI; Wang, 2013), posterior-weighted CDM discrimination index (PWCDI; Zheng & Chang, 2016) and posterior-weighted attribute-level CDM discrimination index (PWACDI; Zheng & Chang, 2016). The impact of Q-matrix structure, item quality, and test termination rules on the efficiency of item selection algorithms is also investigated. Evaluation measures include the attribute classification accuracy (fixed-length experiment) and test length of CDCAT until stopping (variable-length experiment).

The results of the study indicate that the efficiency of item selection is improved by directly modeling the attribute hierarchies using CBN. The test length until achieving diagnosis probability threshold was reduced to 50-70% for CBN based CAT compared to the CD-CAT assuming independence of attributes. The magnitude of improvement is greater when the cognitive model of the test includes more attributes and when the test length is shorter. We conclude by discussing how Q-matrix structure, item quality, and test termination rules affect the efficiency.

References

Beerenwinkel, N., Eriksson, N., & Sturmfels, B. (2007). Conjunctive bayesian networks. Bernoulli, 893- 909.

Chen, P., Xin, T., Wang, C., & Chang, H. H. (2012). Online calibration methods for the DINA model with independent attributes in CD-CAT. Psychometrika, 77(2), 201-222.

Cheng, Y. (2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74(4), 619-632.

Culbertson, M. J. (2016). Bayesian networks in educational assessment: the state of the field. Applied Psychological Measurement, 40(1), 3-21.

Kaplan, M., de la Torre, J., & Barrada, J. R. (2015). New item selection methods for cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 39(3), 167-188.

Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The attribute hierarchy method for cognitive assessment: a variation on Tatsuoka's rule‐space approach. Journal of Educational Measurement, 41(3), 205-237.

Tatsuoka, C. (2002). Data analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society: Series C (Applied Statistics), 51(3), 337-350.

Templin, J., & Bradshaw, L. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79(2), 317-339. Wang, C. (2013). Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length. Educational and Psychological Measurement, 73(6), 1017-1035.

Wang, C., Chang, H. H., & Huebner, A. (2011). Restrictive stochastic item selection methods in cognitive diagnostic computerized adaptive testing. Journal of Educational Measurement, 48(3), 255-273.

Xu, X., Chang, H., & Douglas, J. (2003, April). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of National Council on Measurement in Education, Chicago.

Zheng, C., & Chang, H. H. (2016). High-efficiency response distribution–based item selection algorithms for short-length cognitive diagnostic computerized adaptive testing. Applied Psychological Measurement, 40(8), 608-624.

Session Video

10aCD-CAT10aConjuctive Bayesian Network Modeling10aitem selection1 aHan, Soo-Yun1 aYoo, Yun, Joo uhttps://drive.google.com/open?id=1RbO2gd4aULqsSgRi_VZudNN_edX82NeD02108nas a2200169 4500008004100000245005200041210005100093260005500144520156900199653002101768653000801789653002301797100001501820700001901835700001301854856007101867 2017 eng d00aMHK-MST Design and the Related Simulation Study0 aMHKMST Design and the Related Simulation Study aNiigata, JapanbNiigata Seiryo Universityc08/20173 a

The MHK is a national standardized exam that tests and rates Chinese language proficiency. It assesses non-native Chinese minorities’ abilities in using the Chinese language in their daily, academic and professional lives; Computerized multistage adaptive testing (MST) is a combination of conventional paper-and-pencil (P&P) and item level computerized adaptive test (CAT), it is a kind of test forms based on computerized technology, take the item set as the scoring unit. It can be said that, MST estimate the Ability extreme value more accurate than conventional paper-and-pencil (P&P), also used the CAT auto-adapted characteristic to reduce the examination length and the score time of report. At present, MST has used in some large test, like Uniform CPA Examination and Graduate Record Examination(GRE). Therefore, it is necessary to develop the MST of application in China.

Based on consideration of the MHK characteristics and its future development, the researchers start with design of MHK-MST. This simulation study is conducted to validate the performance of the MHK -MST system. Real difficulty parameters of MHK items and the simulated ability parameters of the candidates are used to generate the original score matrix and the item modules are delivered to the candidates following the adaptive procedures set according to the path rules. This simulation study provides a sound basis for the implementation of MHK-MST.

Session Video

10alanguage testing10aMHK10amultistage testing1 aYuyu, Ling1 aChenglin, Zhou1 aJie, Ren uhttp://www.iacat.org/mhk-mst-design-and-related-simulation-study-001840nas a2200145 4500008004100000245010900041210006900150260005500219520128100274100001501555700001401570700001901584700002001603856007101623 2017 eng d00aA Simulation Study to Compare Classification Method in Cognitive Diagnosis Computerized Adaptive Testing0 aSimulation Study to Compare Classification Method in Cognitive D aNiigata, JapanbNiigata Seiryo Universityc08/20173 a

Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT) combines the strengths of both CAT and cognitive diagnosis. Cognitive diagnosis models that can be viewed as restricted latent class models have been developed to classify the examinees into the correct profile of skills that have been mastered and those that have not so as to get more efficient remediation. Chiu & Douglas (2013) introduces a nonparametric procedure that only requires specification of Q-matrix to classify by proximity to ideal response pattern. In this article, we compare nonparametric procedure with common profile estimation method like maximum a posterior (MAP) in CD-CAT. Simulation studies consider a variety of Q-matrix structure, the number of attributes, ways to generate attribute profiles, and item quality. Results indicate that nonparametric procedure consistently gets the higher pattern and attribute recovery rate in nearly all conditions.

References

Chiu, C.-Y., & Douglas, J. (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response patterns. Journal of Classification, 30, 225-250. doi: 10.1007/s00357-013-9132-9

Session Video

1 aYang, Jing1 aTao, Jian1 aChang, Hua-Hua1 aShi, Ning-Zhong uhttps://drive.google.com/open?id=1jCL3fPZLgzIdwvEk20D-FliZ15OTUtpr01505nas a2200157 4500008003900000022001400039245008900053210006900142300001200211490000700223520101800230100001701248700001501265700002601280856004101306 2015 d a1745-398400aA Comparison of IRT Proficiency Estimation Methods Under Adaptive Multistage Testing0 aComparison of IRT Proficiency Estimation Methods Under Adaptive  a70–790 v523 aThis inquiry is an investigation of item response theory (IRT) proficiency estimators’ accuracy under multistage testing (MST). We chose a two-stage MST design that includes four modules (one at Stage 1, three at Stage 2) and three difficulty paths (low, middle, high). We assembled various two-stage MST panels (i.e., forms) by manipulating two assembly conditions in each module, such as difficulty level and module length. For each panel, we investigated the accuracy of examinees’ proficiency levels derived from seven IRT proficiency estimators. The choice of Bayesian (prior) versus non-Bayesian (no prior) estimators was of more practical significance than the choice of number-correct versus item-pattern scoring estimators. The Bayesian estimators were slightly more efficient than the non-Bayesian estimators, resulting in smaller overall error. Possible score changes caused by the use of different proficiency estimators would be nonnegligible, particularly for low- and high-performing examinees.1 aKim, Sooyeon1 aMoses, Tim1 aYoo, Hanwook, (Henry) uhttp://dx.doi.org/10.1111/jedm.1206301322nas a2200145 4500008003900000245005100039210004900090300000900139490000700148520091500155100001801070700001801088700001901106856005101125 2015 d00aOnline Item Calibration for Q-Matrix in CD-CAT0 aOnline Item Calibration for QMatrix in CDCAT a5-150 v393 a

Item replenishment is important for maintaining a large-scale item bank. In this article, the authors consider calibrating new items based on pre-calibrated operational items under the deterministic inputs, noisy-and-gate model, the specification of which includes the so-called -matrix, as well as the slipping and guessing parameters. Making use of the maximum likelihood and Bayesian estimators for the latent knowledge states, the authors propose two methods for the calibration. These methods are applicable to both traditional paper–pencil–based tests, for which the selection of operational items is prefixed, and computerized adaptive tests, for which the selection of operational items is sequential and random. Extensive simulations are done to assess and to compare the performance of these approaches. Extensions to other diagnostic classification models are also discussed.

1 aChen, Yunxiao1 aLiu, Jingchen1 aYing, Zhiliang uhttp://apm.sagepub.com/content/39/1/5.abstract00491nam a2200133 4500008004100000020002500041245006100066210006000127260002900187100001600216700002500232700001900257856008100276 2014 eng d a13-978-1-4665-0577-300aComputerized multistage testing: Theory and applications0 aComputerized multistage testing Theory and applications aBoca Raton FLbCRC Press1 aYan, Duanli1 avon Davier, Alina, A1 aLewis, Charles uhttp://www.iacat.org/computerized-multistage-testing-theory-and-applications02104nas a2200133 4500008003900000022001400039245013800053210006900191300001200260490000700272520163500279100001501914856004101929 2014 d a1745-398400aMultidimensional CAT Item Selection Methods for Domain Scores and Composite Scores With Item Exposure Control and Content Constraints0 aMultidimensional CAT Item Selection Methods for Domain Scores an a18–380 v513 a

The intent of this research was to find an item selection procedure in the multidimensional computer adaptive testing (CAT) framework that yielded higher precision for both the domain and composite abilities, had a higher usage of the item pool, and controlled the exposure rate. Five multidimensional CAT item selection procedures (minimum angle; volume; minimum error variance of the linear combination; minimum error variance of the composite score with optimized weight; and Kullback-Leibler information) were studied and compared with two methods for item exposure control (the Sympson-Hetter procedure and the fixed-rate procedure, the latter simply refers to putting a limit on the item exposure rate) using simulated data. The maximum priority index method was used for the content constraints. Results showed that the Sympson-Hetter procedure yielded better precision than the fixed-rate procedure but had much lower item pool usage and took more time. The five item selection procedures performed similarly under Sympson-Hetter. For the fixed-rate procedure, there was a trade-off between the precision of the ability estimates and the item pool usage: the five procedures had different patterns. It was found that (1) Kullback-Leibler had better precision but lower item pool usage; (2) minimum angle and volume had balanced precision and item pool usage; and (3) the two methods minimizing the error variance had the best item pool usage and comparable overall score recovery but less precision for certain domains. The priority index for content constraints and item exposure was implemented successfully.

1 aYao, Lihua uhttp://dx.doi.org/10.1111/jedm.1203201548nas a2200145 4500008003900000245008200039210006900121300001200190490000700202520108300209100001501292700002001307700002201327856005301349 2014 d00aUsing Multidimensional CAT to Administer a Short, Yet Precise, Screening Test0 aUsing Multidimensional CAT to Administer a Short Yet Precise Scr a614-6310 v383 a

Multidimensional computerized adaptive testing (MCAT) provides a mechanism by which the simultaneous goals of accurate prediction and minimal testing time for a screening test could both be met. This article demonstrates the use of MCAT to administer a screening test for the Computerized Adaptive Testing–Armed Services Vocational Aptitude Battery (CAT-ASVAB) under a variety of manipulated conditions. CAT-ASVAB is a test battery administered via unidimensional CAT (UCAT) that is used to qualify applicants for entry into the U.S. military and assign them to jobs. The primary research question being evaluated is whether the use of MCAT to administer a screening test can lead to significant reductions in testing time from the full-length selection test, without significant losses in score precision. Different stopping rules, item selection methods, content constraints, time constraints, and population distributions for the MCAT administration are evaluated through simulation, and compared with results from a regular full-length UCAT administration.

1 aYao, Lihua1 aPommerich, Mary1 aSegall, Daniel, O uhttp://apm.sagepub.com/content/38/8/614.abstract02024nas a2200121 4500008003900000245011000039210006900149300000900218490000700227520160200234100001501836856005101851 2013 d00aComparing the Performance of Five Multidimensional CAT Selection Procedures With Different Stopping Rules0 aComparing the Performance of Five Multidimensional CAT Selection a3-230 v373 a

Through simulated data, five multidimensional computerized adaptive testing (MCAT) selection procedures with varying test lengths are examined and compared using different stopping rules. Fixed item exposure rates are used for all the items, and the Priority Index (PI) method is used for the content constraints. Two stopping rules, standard error (SE) and predicted standard error reduction (PSER), are proposed; each MCAT selection process is stopped if either the required precision has been achieved or the selected number of items has reached the maximum limit. The five procedures are as follows: minimum angle (Ag), volume (Vm), minimize the error variance of the linear combination (V 1), minimize the error variance of the composite score with the optimized weight (V 2), and Kullback–Leibler (KL) information. The recovery for the domain scores or content scores and their overall score, test length, and test reliability are compared across the five MCAT procedures and between the two stopping rules. It is found that the two stopping rules are implemented successfully and that KL uses the least number of items to reach the same precision level, followed by Vm; Ag uses the largest number of items. On average, to reach a precision of SE = .35, 40, 55, 63, 63, and 82 items are needed for KL, Vm, V 1, V 2, and Ag, respectively, for the SE stopping rule. PSER yields 38, 45, 53, 58, and 68 items for KL, Vm, V 1, V 2, and Ag, respectively; PSER yields only slightly worse results than SE, but with much fewer items. Overall, KL is recommended for varying-length MCAT.

1 aYao, Lihua uhttp://apm.sagepub.com/content/37/1/3.abstract01409nas a2200157 4500008003900000245009300039210006900132300001000201490000700211520091300218100002301131700001501154700001701169700001301186856005201199 2013 d00aThe Influence of Item Calibration Error on Variable-Length Computerized Adaptive Testing0 aInfluence of Item Calibration Error on VariableLength Computeriz a24-400 v373 a

Variable-length computerized adaptive testing (VL-CAT) allows both items and test length to be “tailored” to examinees, thereby achieving the measurement goal (e.g., scoring precision or classification) with as few items as possible. Several popular test termination rules depend on the standard error of the ability estimate, which in turn depends on the item parameter values. However, items are chosen on the basis of their parameter estimates, and capitalization on chance may occur. In this article, the authors investigated the effects of capitalization on chance on test length and classification accuracy in several VL-CAT simulations. The results confirm that capitalization on chance occurs in VL-CAT and has complex effects on test length, ability estimation, and classification accuracy. These results have important implications for the design and implementation of VL-CATs.

1 aPatton, Jeffrey, M1 aYing Cheng1 aYuan, Ke-Hai1 aDiao, Qi uhttp://apm.sagepub.com/content/37/1/24.abstract01556nas a2200169 4500008003900000245012400039210006900163300001000232490000700242520099400249100001901243700001801262700001801280700001601298700002001314856005201334 2012 d00aAn Empirical Evaluation of the Slip Correction in the Four Parameter Logistic Models With Computerized Adaptive Testing0 aEmpirical Evaluation of the Slip Correction in the Four Paramete a75-870 v363 a

In a selected response test, aberrant responses such as careless errors and lucky guesses might cause error in ability estimation because these responses do not actually reflect the knowledge that examinees possess. In a computerized adaptive test (CAT), these aberrant responses could further cause serious estimation error due to dynamic item administration. To enhance the robust performance of CAT against aberrant responses, Barton and Lord proposed the four-parameter logistic (4PL) item response theory (IRT) model. However, most studies relevant to the 4PL IRT model were conducted based on simulation experiments. This study attempts to investigate the performance of the 4PL IRT model as a slip-correction mechanism with an empirical experiment. The results showed that the 4PL IRT model could not only reduce the problematic underestimation of the examinees’ ability introduced by careless mistakes in practical situations but also improve measurement efficiency.

1 aYen, Yung-Chin1 aHo, Rong-Guey1 aLaio, Wen-Wei1 aChen, Li-Ju1 aKuo, Ching-Chin uhttp://apm.sagepub.com/content/36/2/75.abstract02592nas a2200313 4500008004100000020004600041245011200087210006900199250001500268260001900283300000700302490000600309520160100315100001901916700001801935700001701953700001801970700001701988700001402005700001602019700001502035700001402050700002602064700001602090700001302106700001502119700001902134856012502153 2011 Eng d a1477-7525 (Electronic)1477-7525 (Linking)00aCross-cultural development of an item list for computer-adaptive testing of fatigue in oncological patients0 aCrosscultural development of an item list for computeradaptive t a2011/03/31 cMarch 29, 2011 a100 v93 aABSTRACT: INTRODUCTION: Within an ongoing project of the EORTC Quality of Life Group, we are developing computerized adaptive test (CAT) measures for the QLQ-C30 scales. These new CAT measures are conceptualised to reflect the same constructs as the QLQ-C30 scales. Accordingly, the Fatigue-CAT is intended to capture physical and general fatigue. METHODS: The EORTC approach to CAT development comprises four phases (literature search, operationalisation, pre-testing, and field testing). Phases I-III are described in detail in this paper. A literature search for fatigue items was performed in major medical databases. After refinement through several expert panels, the remaining items were used as the basis for adapting items and/or formulating new items fitting the EORTC item style. To obtain feedback from patients with cancer, these English items were translated into Danish, French, German, and Spanish and tested in the respective countries. RESULTS: Based on the literature search a list containing 588 items was generated. After a comprehensive item selection procedure focusing on content, redundancy, item clarity and item difficulty a list of 44 fatigue items was generated. Patient interviews (n=52) resulted in 12 revisions of wording and translations. DISCUSSION: The item list developed in phases I-III will be further investigated within a field-testing phase (IV) to examine psychometric characteristics and to fit an item response theory model. The Fatigue CAT based on this item bank will provide scores that are backward-compatible to the original QLQ-C30 fatigue scale.1 aGiesinger, J M1 aPetersen, M A1 aGroenvold, M1 aAaronson, N K1 aArraras, J I1 aConroy, T1 aGamper, E M1 aKemmler, G1 aKing, M T1 aOberguggenberger, A S1 aVelikova, G1 aYoung, T1 aHolzner, B1 aEortc-Qlg, E O uhttp://www.iacat.org/content/cross-cultural-development-item-list-computer-adaptive-testing-fatigue-oncological-patients00557nas a2200157 4500008004100000245008700041210006900128300001400197490001000211100001300221700001200234700001400246700001400260700001400274856011100288 2010 eng d00aDevelopment and evaluation of a confidence-weighting computerized adaptive testing0 aDevelopment and evaluation of a confidenceweighting computerized a163–1760 v13(3)1 aYen, Y C1 aHo, R G1 aChen, L J1 aChou, K Y1 aChen, Y L uhttp://www.iacat.org/content/development-and-evaluation-confidence-weighting-computerized-adaptive-testing03103nas a2200445 4500008004100000020004100041245012000082210006900202250001500271260001000286300001100296490000700307520175400314653003802068653002102106653001002127653000902137653002202146653002802168653003302196653001102229653001102240653000902251653001602260653001802276653001902294653003102313653003102344653001602375100001602391700001002407700001402417700001502431700001402446700001502460700001802475700002402493700001802517856012202535 2010 eng d a0161-8105 (Print)0161-8105 (Linking)00aDevelopment and validation of patient-reported outcome measures for sleep disturbance and sleep-related impairments0 aDevelopment and validation of patientreported outcome measures f a2010/06/17 cJun 1 a781-920 v333 aSTUDY OBJECTIVES: To develop an archive of self-report questions assessing sleep disturbance and sleep-related impairments (SRI), to develop item banks from this archive, and to validate and calibrate the item banks using classic validation techniques and item response theory analyses in a sample of clinical and community participants. DESIGN: Cross-sectional self-report study. SETTING: Academic medical center and participant homes. PARTICIPANTS: One thousand nine hundred ninety-three adults recruited from an Internet polling sample and 259 adults recruited from medical, psychiatric, and sleep clinics. INTERVENTIONS: None. MEASUREMENTS AND RESULTS: This study was part of PROMIS (Patient-Reported Outcomes Information System), a National Institutes of Health Roadmap initiative. Self-report item banks were developed through an iterative process of literature searches, collecting and sorting items, expert content review, qualitative patient research, and pilot testing. Internal consistency, convergent validity, and exploratory and confirmatory factor analysis were examined in the resulting item banks. Factor analyses identified 2 preliminary item banks, sleep disturbance and SRI. Item response theory analyses and expert content review narrowed the item banks to 27 and 16 items, respectively. Validity of the item banks was supported by moderate to high correlations with existing scales and by significant differences in sleep disturbance and SRI scores between participants with and without sleep disorders. CONCLUSIONS: The PROMIS sleep disturbance and SRI item banks have excellent measurement properties and may prove to be useful for assessing general aspects of sleep and SRI with various groups of patients and interventions.10a*Outcome Assessment (Health Care)10a*Self Disclosure10aAdult10aAged10aAged, 80 and over10aCross-Sectional Studies10aFactor Analysis, Statistical10aFemale10aHumans10aMale10aMiddle Aged10aPsychometrics10aQuestionnaires10aReproducibility of Results10aSleep Disorders/*diagnosis10aYoung Adult1 aBuysse, D J1 aYu, L1 aMoul, D E1 aGermain, A1 aStover, A1 aDodds, N E1 aJohnston, K L1 aShablesky-Cade, M A1 aPilkonis, P A uhttp://www.iacat.org/content/development-and-validation-patient-reported-outcome-measures-sleep-disturbance-and-sleep02223nas a2200289 4500008004100000020004600041245010800087210006900195250001500264300001200279490000700291520131000298100001801608700001701626700001801643700001401661700001401675700001801689700001401707700001701721700001501738700001301753700001401766700001601780700001301796856012401809 2010 Eng d a1573-2649 (Electronic)0962-9343 (Linking)00aDevelopment of computerized adaptive testing (CAT) for the EORTC QLQ-C30 physical functioning dimension0 aDevelopment of computerized adaptive testing CAT for the EORTC Q a2010/10/26 a479-4900 v203 aPURPOSE: Computerized adaptive test (CAT) methods, based on item response theory (IRT), enable a patient-reported outcome instrument to be adapted to the individual patient while maintaining direct comparability of scores. The EORTC Quality of Life Group is developing a CAT version of the widely used EORTC QLQ-C30. We present the development and psychometric validation of the item pool for the first of the scales, physical functioning (PF). METHODS: Initial developments (including literature search and patient and expert evaluations) resulted in 56 candidate items. Responses to these items were collected from 1,176 patients with cancer from Denmark, France, Germany, Italy, Taiwan, and the United Kingdom. The items were evaluated with regard to psychometric properties. RESULTS: Evaluations showed that 31 of the items could be included in a unidimensional IRT model with acceptable fit and good content coverage, although the pool may lack items at the upper extreme (good PF). There were several findings of significant differential item functioning (DIF). However, the DIF findings appeared to have little impact on the PF estimation. CONCLUSIONS: We have established an item pool for CAT measurement of PF and believe that this CAT instrument will clearly improve the EORTC measurement of PF.1 aPetersen, M A1 aGroenvold, M1 aAaronson, N K1 aChie, W C1 aConroy, T1 aCostantini, A1 aFayers, P1 aHelbostad, J1 aHolzner, B1 aKaasa, S1 aSinger, S1 aVelikova, G1 aYoung, T uhttp://www.iacat.org/content/development-computerized-adaptive-testing-cat-eortc-qlq-c30-physical-functioning-dimension01417nas a2200121 4500008003900000245015400039210006900193300001200262490000700274520094600281100001501227856005301242 2010 d00aImproving Cognitive Diagnostic Computerized Adaptive Testing by Balancing Attribute Coverage: The Modified Maximum Global Discrimination Index Method0 aImproving Cognitive Diagnostic Computerized Adaptive Testing by  a902-9130 v703 a

This article proposes a new item selection method, namely, the modified maximum global discrimination index (MMGDI) method, for cognitive diagnostic computerized adaptive testing (CD-CAT). The new method captures two aspects of the appeal of an item: (a) the amount of contribution it can make toward adequate coverage of every attribute and (b) the amount of contribution it can make toward recovering the latent cognitive profile. A simulation study shows that the new method ensures adequate coverage of every attribute, which improves the validity of the test scores, and defensibility of the proposed uses of the test. Furthermore, compared with the original global discrimination index method, the MMGDI method improves the recovery rate of each attribute and of the entire cognitive profile, especially the latter. Therefore, the new method improves both the validity and reliability of the test scores from a CD-CAT program.

1 aYing Cheng uhttp://epm.sagepub.com/content/70/6/902.abstract01651nas a2200157 4500008003900000245010700039210006900146300001000215490000700225520113900232100001501371700001901386700002101405700001501426856005201441 2009 d00aConstraint-Weighted a-Stratification for Computerized Adaptive Testing With Nonstatistical Constraints0 aConstraintWeighted aStratification for Computerized Adaptive Tes a35-490 v693 a

a-stratification is a method that utilizes items with small discrimination (a) parameters early in an exam and those with higher a values when more is learned about the ability parameter. It can achieve much better item usage than the maximum information criterion (MIC). To make a-stratification more practical and more widely applicable, a method for weighting the item selection process in a-stratification as a means of satisfying multiple test constraints is proposed. This method is studied in simulation against an analogous method without stratification as well as a-stratification using descending-rather than ascending-a procedures. In addition, a variation of a-stratification that allows for unbalanced usage of a parameters is included in the study to examine the trade-off between efficiency and exposure control. Finally, MIC and randomized item selection are included as baseline measures. Results indicate that the weighting mechanism successfully addresses the constraints, that stratification helps to a great extent balancing exposure rates, and that the ascending-a design improves measurement precision.

1 aYing Cheng1 aChang, Hua-Hua1 aDouglas, Jeffrey1 aFanmin Guo uhttp://epm.sagepub.com/content/69/1/35.abstract01669nas a2200169 4500008003900000245009500039210007100134300000900205490000700214520113900221100001701360700001401377700002201391700001901413700001601432856005101448 2008 d00aComparability of Computer-Based and Paper-and-Pencil Testing in K–12 Reading Assessments0 aComparability of ComputerBased and PaperandPencil Testing in K–1 a5-240 v683 a

In recent years, computer-based testing (CBT) has grown in popularity, is increasingly being implemented across the United States, and will likely become the primary mode for delivering tests in the future. Although CBT offers many advantages over traditional paper-and-pencil testing, assessment experts, researchers, practitioners, and users have expressed concern about the comparability of scores between the two test administration modes. To help provide an answer to this issue, a meta-analysis was conducted to synthesize the administration mode effects of CBTs and paper-and-pencil tests on K—12 student reading assessments. Findings indicate that the administration mode had no statistically significant effect on K—12 student reading achievement scores. Four moderator variables—study design, sample size, computer delivery algorithm, and computer practice—made statistically significant contributions to predicting effect size. Three moderator variables—grade level, type of test, and computer delivery method—did not affect the differences in reading scores between test modes.

1 aShudong Wang1 aHong Jiao1 aYoung, Michael, J1 aBrooks, Thomas1 aOlson, John uhttp://epm.sagepub.com/content/68/1/5.abstract01427nas a2200133 4500008003900000245008900039210006900128300001200197490000700209520094400216100001601160700001201176856010501188 2008 d00aTo Weight Or Not To Weight? Balancing Influence Of Initial Items In Adaptive Testing0 aTo Weight Or Not To Weight Balancing Influence Of Initial Items  a441-4500 v733 a

It has been widely reported that in computerized adaptive testing some examinees may get much lower scores than they would normally if an alternative paper-and-pencil version were given. The main purpose of this investigation is to quantitatively reveal the cause for the underestimation phenomenon. The logistic models, including the 1PL, 2PL, and 3PL models, are used to demonstrate our assertions. Our analytical derivation shows that, under the maximum information item selection strategy, if an examinee failed a few items at the beginning of the test, easy but more discriminating items are likely to be administered. Such items are ineffective to move the estimate close to the true theta, unless the test is sufficiently long or a variable-length test is used. Our results also indicate that a certain weighting mechanism is necessary to make the algorithm rely less on the items administered at the beginning of the test.

1 aChang, H -H1 aYing, Z uhttp://www.iacat.org/content/weight-or-not-weight-balancing-influence-initial-items-adaptive-testing01227nas a2200229 4500008004100000245014700041210006900188300001100257490000700268520046800275100001300743700001300756700001600769700001600785700001300801700001300814700001200827700001500839700001300854700001200867856011800879 2007 eng d00aThe Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years0 aPatientReported Outcomes Measurement Information System PROMIS p aS3-S110 v453 aThe National Institutes of Health (NIH) Patient-Reported Outcomes Measurement Information System (PROMIS) Roadmap initiative (www.nihpromis.org) is a 5-year cooperative group program of research designed to develop, validate, and standardize item banks to measure patient-reported outcomes (PROs) relevant across common medical conditions. In this article, we will summarize the organization and scientific activity of the PROMIS network during its first 2 years.1 aCella, D1 aYount, S1 aRothrock, N1 aGershon, RC1 aCook, KF1 aReeve, B1 aAder, D1 aFries, J F1 aBruce, B1 aRose, M uhttp://www.iacat.org/content/patient-reported-outcomes-measurement-information-system-promis-progress-nih-roadmap00482nas a2200133 4500008004100000245007700041210006900118300001400187490000600201100001300207700001900220700001000239856009900249 2007 eng d00aTwo-phase item selection procedure for flexible content balancing in CAT0 aTwophase item selection procedure for flexible content balancing a467–4820 v31 aCheng, Y1 aChang, Hua-Hua1 aYi, Q uhttp://www.iacat.org/content/two-phase-item-selection-procedure-flexible-content-balancing-cat01435nas a2200145 4500008003900000245007700039210006900116300001200185490000700197520098600204100001501190700001901205700001201224856005301236 2007 d00aTwo-Phase Item Selection Procedure for Flexible Content Balancing in CAT0 aTwoPhase Item Selection Procedure for Flexible Content Balancing a467-4820 v313 a

Content balancing is an important issue in the design and implementation of computerized adaptive testing (CAT). Content-balancing techniques that have been applied in fixed content balancing, where the number of items from each content area is fixed, include constrained CAT (CCAT), the modified multinomial model (MMM), modified constrained CAT (MCCAT), and others. In this article, four methods are proposed to address the flexible content-balancing issue with the a-stratification design, named STR_C. The four methods are MMM+, an extension of MMM; MCCAT+, an extension of MCCAT; the TPM method, a two-phase content-balancing method using MMM in both phases; and the TPF method, a two-phase content-balancing method using MMM in the first phase and MCCAT in the second. Simulation results show that all of the methods work well in content balancing, and TPF performs the best in item exposure control and item pool utilization while maintaining measurement precision.

1 aYing Cheng1 aChang, Hua-Hua1 aQing Yi uhttp://apm.sagepub.com/content/31/6/467.abstract00373nas a2200121 4500008004100000245004100041210004100082300001200123490001000135100001700145700001900162856007000181 2006 eng d00aAssessing CAT Test Security Severity0 aAssessing CAT Test Security Severity a62–630 v30(1)1 aYi, Zhang, J1 aChang, Hua-Hua uhttp://www.iacat.org/content/assessing-cat-test-security-severity01947nas a2200241 4500008004100000020002200041245011200063210006900175260004100244300001200285490000700297520102300304653003401327653002401361653004701385653002401432653002101456653007001477100001301547700001401560700001001574856012101584 2006 eng d a0022-0655 (Print)00aComparing methods of assessing differential item functioning in a computerized adaptive testing environment0 aComparing methods of assessing differential item functioning in  bBlackwell Publishing: United Kingdom a245-2640 v433 aMantel-Haenszel and SIBTEST, which have known difficulty in detecting non-unidirectional differential item functioning (DIF), have been adapted with some success for computerized adaptive testing (CAT). This study adapts logistic regression (LR) and the item-response-theory-likelihood-ratio test (IRT-LRT), capable of detecting both unidirectional and non-unidirectional DIF, to the CAT environment in which pretest items are assumed to be seeded in CATs but not used for trait estimation. The proposed adaptation methods were evaluated with simulated data under different sample size ratios and impact conditions in terms of Type I error, power, and specificity in identifying the form of DIF. The adapted LR and IRT-LRT procedures are more powerful than the CAT version of SIBTEST for non-unidirectional DIF detection. The good Type I error control provided by IRT-LRT under extremely unequal sample sizes and large impact is encouraging. Implications of these and other findings are discussed. all rights reserved)10acomputerized adaptive testing10aeducational testing10aitem response theory likelihood ratio test10alogistic regression10atrait estimation10aunidirectional & non-unidirectional differential item functioning1 aLei, P-W1 aChen, S-Y1 aYu, L uhttp://www.iacat.org/content/comparing-methods-assessing-differential-item-functioning-computerized-adaptive-testing01526nas a2200157 4500008003900000022001400039245011200053210006900165300001400234490000700248520101100255100001601266700001901282700001201301856005501313 2006 d a1745-398400aComparing Methods of Assessing Differential Item Functioning in a Computerized Adaptive Testing Environment0 aComparing Methods of Assessing Differential Item Functioning in  a245–2640 v433 a

Mantel-Haenszel and SIBTEST, which have known difficulty in detecting non-unidirectional differential item functioning (DIF), have been adapted with some success for computerized adaptive testing (CAT). This study adapts logistic regression (LR) and the item-response-theory-likelihood-ratio test (IRT-LRT), capable of detecting both unidirectional and non-unidirectional DIF, to the CAT environment in which pretest items are assumed to be seeded in CATs but not used for trait estimation. The proposed adaptation methods were evaluated with simulated data under different sample size ratios and impact conditions in terms of Type I error, power, and specificity in identifying the form of DIF. The adapted LR and IRT-LRT procedures are more powerful than the CAT version of SIBTEST for non-unidirectional DIF detection. The good Type I error control provided by IRT-LRT under extremely unequal sample sizes and large impact is encouraging. Implications of these and other findings are discussed.

1 aLei, Pui-Wa1 aChen, Shu-Ying1 aYu, Lan uhttp://dx.doi.org/10.1111/j.1745-3984.2006.00015.x01613nas a2200145 4500008003900000245011900039210006900158300001200227490000700239520110300246100002001349700002001369700002501389856005301414 2006 d00aEffects of Estimation Bias on Multiple-Category Classification With an IRT-Based Adaptive Classification Procedure0 aEffects of Estimation Bias on MultipleCategory Classification Wi a545-5640 v663 a

The effects of five ability estimators, that is, maximum likelihood estimator, weighted likelihood estimator, maximum a posteriori, expected a posteriori, and Owen's sequential estimator, on the performances of the item response theory–based adaptive classification procedure on multiple categories were studied via simulations. The following results were found. (a) The Bayesian estimators were more likely to misclassify examinees into an inward category because of their inward biases, when a fixed start value of zero was assigned to every examinee. (b) When moderately accurate start values were available, however, Bayesian estimators produced classifications that were slightly more accurate than was the maximum likelihood estimator or weighted likelihood estimator. Expected a posteriori was the procedure that produced the most accurate results among the three Bayesian methods. (c) All five estimators produced equivalent efficiencies in terms of number of items required, which was 50 or more items except for abilities that were less than -2.00 or greater than 2.00.

1 aYang, Xiangdong1 aPoggio, John, C1 aGlasnapp, Douglas, R uhttp://epm.sagepub.com/content/66/4/545.abstract00498nas a2200121 4500008004100000245010000041210006900141300001200210490001000222100000700232700001400239856012300253 2005 eng d00aDesign and evaluation of an XML-based platform-independent computerized adaptive testing system0 aDesign and evaluation of an XMLbased platformindependent compute a230-2370 v48(2)1 aHo1 aYen, Y -C uhttp://www.iacat.org/content/design-and-evaluation-xml-based-platform-independent-computerized-adaptive-testing-system00452nas a2200121 4500008004100000245006700041210006700108260002100175100001000196700001300206700001900219856009200238 2005 eng d00aIdentifying practical indices for enhancing item pool security0 aIdentifying practical indices for enhancing item pool security aMontreal, Canada1 aYi, Q1 aZhang, J1 aChang, Hua-Hua uhttp://www.iacat.org/content/identifying-practical-indices-enhancing-item-pool-security01341nas a2200145 4500008003900000245008200039210006900121300001200190490000700202520087300209100001601082700001901098700002101117856005701138 2004 d00aAdaptive Testing With Regression Trees in the Presence of Multidimensionality0 aAdaptive Testing With Regression Trees in the Presence of Multid a293-3160 v293 a

It is unrealistic to suppose that standard item response theory (IRT) models will be appropriate for all the new and currently considered computer-based tests. In addition to developing new models, we also need to give attention to the possibility of constructing and analyzing new tests without the aid of strong models. Computerized adaptive testing currently relies heavily on IRT. Alternative, empirically based, nonparametric adaptive testing algorithms exist, but their properties are little known. This article introduces a nonparametric, tree-based algorithm for adaptive testing and shows that it may be superior to conventional, IRT-based adaptive testing in cases where the IRT assumptions are not satisfied. In particular, it shows that the tree-based approach clearly outperformed (one-dimensional) IRT when the pool was strongly two-dimensional.

1 aYan, Duanli1 aLewis, Charles1 aStocking, Martha uhttp://jeb.sagepub.com/cgi/content/abstract/29/3/29301774nas a2200217 4500008004100000245007700041210006900118300001100187490000600198520108400204653001501288653002501303653001601328653001001344653001801354653002101372653003101393100001901424700001501443856009801458 2004 eng d00aPre-equating: a simulation study based on a large scale assessment model0 aPreequating a simulation study based on a large scale assessment a301-180 v53 aAlthough post-equating (PE) has proven to be an acceptable method in the scaling and equating of items and forms, there are times when the turn-around period for equating and converting raw scores to scale scores is so small that PE cannot be undertaken within the prescribed time frame. In such cases, pre-equating (PrE) could be considered as an acceptable alternative. Assessing the feasibility of using item calibrations from the item bank (as in PrE) is conditioned on the equivalency of the calibrations and the errors associated with it vis a vis the results obtained via PE. This paper creates item banks over three periods of item introduction into the banks and uses the Rasch model in examining data with respect to the recovery of item parameters, the measurement error, and the effect cut-points have on examinee placement in both the PrE and PE situations. Results indicate that PrE is a viable solution to PE provided the stability of the item calibrations are enhanced by using large sample sizes (perhaps as large as full-population) in populating the item bank.10a*Databases10a*Models, Theoretical10aCalibration10aHuman10aPsychometrics10aReference Values10aReproducibility of Results1 aTaherbhai, H M1 aYoung, M J uhttp://www.iacat.org/content/pre-equating-simulation-study-based-large-scale-assessment-model00411nas a2200121 4500008003900000245006100039210005900100300001400159490000700173100001000180700001600190856008300206 2003 d00aa-Stratified multistage CAT design with content-blocking0 aaStratified multistage CAT design with contentblocking a359–3780 v561 aYi, Q1 aChang, H -H uhttp://www.iacat.org/content/stratified-multistage-cat-design-content-blocking00468nas a2200133 4500008004100000245006800041210006800109260001500177100001000192700001600202700001200218700001300230856009100243 2003 eng d00aEffects of test administration mode on item parameter estimates0 aEffects of test administration mode on item parameter estimates aChicago IL1 aYi, Q1 aHarris, D J1 aWang, T1 aBan, J-C uhttp://www.iacat.org/content/effects-test-administration-mode-item-parameter-estimates00538nas a2200121 4500008004100000245013200041210006900173260001500242100001000257700001200267700001200279856012500291 2003 eng d00aImplementing the a-stratified method with b blocking in computerized adaptive testing with the generalized partial credit model0 aImplementing the astratified method with b blocking in computeri aChicago IL1 aYi, Q1 aWang, T1 aWang, S uhttp://www.iacat.org/content/implementing-stratified-method-b-blocking-computerized-adaptive-testing-generalized-partial00500nas a2200109 4500008004100000245010900041210006900150260001500219100001900234700001200253856012500265 2003 eng d00aTest-score comparability, ability estimation, and item-exposure control in computerized adaptive testing0 aTestscore comparability ability estimation and itemexposure cont aChicago IL1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/test-score-comparability-ability-estimation-and-item-exposure-control-computerized-adaptive00511nas a2200109 4500008004100000245012600041210006900167260001900236100001200255700001200267856012200279 2002 eng d00aAccuracy of the ability estimate and the item exposure rate under multidimensional adaptive testing with item constraints0 aAccuracy of the ability estimate and the item exposure rate unde aNew Orleans LA1 aLi, Y H1 aYu, N Y uhttp://www.iacat.org/content/accuracy-ability-estimate-and-item-exposure-rate-under-multidimensional-adaptive-testing00527nas a2200121 4500008004100000245009600041210006900137260004600206100001100252700001300263700001600276856011300292 2002 eng d00aAdaptive testing without IRT in the presence of multidimensionality (Research Report 02-09)0 aAdaptive testing without IRT in the presence of multidimensional aPrinceton NJ: Educational Testing Service1 aYan, D1 aLewis, C1 aStocking, M uhttp://www.iacat.org/content/adaptive-testing-without-irt-presence-multidimensionality-research-report-02-0901447nas a2200241 4500008004100000245008000041210006900121300001200190490000700202520067300209653003000882653002800912653002500940653002300965653001600988653002201004653002101026100001301047700001601060700001001076700001601086856010301102 2002 eng d00aData sparseness and on-line pretest item calibration-scaling methods in CAT0 aData sparseness and online pretest item calibrationscaling metho a207-2180 v393 aCompared and evaluated 3 on-line pretest item calibration-scaling methods (the marginal maximum likelihood estimate with 1 expectation maximization [EM] cycle [OEM] method, the marginal maximum likelihood estimate with multiple EM cycles [MEM] method, and M. L. Stocking's Method B) in terms of item parameter recovery when the item responses to the pretest items in the pool are sparse. Simulations of computerized adaptive tests were used to evaluate the results yielded by the three methods. The MEM method produced the smallest average total error in parameter estimation, and the OEM method yielded the largest total error (PsycINFO Database Record (c) 2005 APA )10aComputer Assisted Testing10aEducational Measurement10aItem Response Theory10aMaximum Likelihood10aMethodology10aScaling (Testing)10aStatistical Data1 aBan, J-C1 aHanson, B A1 aYi, Q1 aHarris, D J uhttp://www.iacat.org/content/data-sparseness-and-line-pretest-item-calibration-scaling-methods-cat00471nas a2200097 4500008004100000245011200041210006900153260002000222100001000242856012100252 2002 eng d00aIncorporating the Sympson-Hetter exposure control method into the a-stratified method with content blocking0 aIncorporating the SympsonHetter exposure control method into the aNew Orleans, LA1 aYi, Q uhttp://www.iacat.org/content/incorporating-sympson-hetter-exposure-control-method-stratified-method-content-blocking00504nas a2200121 4500008004100000245009700041210006900138260001900207100001100226700001000237700001300247856012200260 2002 eng d00aStatistical indexes for monitoring item behavior under computer adaptive testing environment0 aStatistical indexes for monitoring item behavior under computer  aNew Orleans LA1 aZhu, R1 aYu, F1 aLiu, S M uhttp://www.iacat.org/content/statistical-indexes-monitoring-item-behavior-under-computer-adaptive-testing-environment00471nas a2200109 4500008004100000245008900041210006900130260001900199100001900218700001200237856011200249 2002 eng d00aTo weight or not to weight – balancing influence of initial and later items in CAT0 aTo weight or not to weight balancing influence of initial and la aNew Orleans LA1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/weight-or-not-weight-%E2%80%93-balancing-influence-initial-and-later-items-cat00374nas a2200109 4500008004100000245005000041210004800091260002400139100001000163700001900173856007200192 2001 eng d00aa-stratified CAT design with content-blocking0 aastratified CAT design with contentblocking aKing of Prussia, PA1 aYi, Q1 aChang, Hua-Hua uhttp://www.iacat.org/content/stratified-cat-design-content-blocking01098nas a2200157 4500008004100000020001400041245007400055210006900129300001200198490000700210520058200217100001900799700001200818700001200830856009800842 2001 eng d a0146-621600aa-Stratified multistage computerized adaptive testing with b blocking0 aaStratified multistage computerized adaptive testing with b bloc a333-3410 v253 aChang & Ying’s (1999) computerized adaptive testing item-selection procedure stratifies the item bank according to a parameter values and requires b parameter values to be evenly distributed across all strata. Thus, a and b parameter values must be incorporated into how strata are formed. A refinement is proposed, based on Weiss’ (1973) stratification of items according to b values. Simulation studies using a retired item bank of a Graduate Record Examination test indicate that the new approach improved control of item exposure rates and reduced mean squared errors. 1 aChang, Hua-Hua1 aQian, J1 aYing, Z uhttp://www.iacat.org/content/stratified-multistage-computerized-adaptive-testing-b-blocking-000842nas a2200157 4500008004100000245007400041210006900115300001100184490000700195520030900202653003400511100001900545700001200564700001200576856009600588 2001 eng d00aa-stratified multistage computerized adaptive testing with b blocking0 aastratified multistage computerized adaptive testing with b bloc a333-410 v253 aProposed a refinement, based on the stratification of items developed by D. Weiss (1973), of the computerized adaptive testing item selection procedure of H. Chang and Z. Ying (1999). Simulation studies using an item bank from the Graduate Record Examination show the benefits of the new procedure. (SLD)10acomputerized adaptive testing1 aChang, Hua-Hua1 aQian, J1 aYang, Z uhttp://www.iacat.org/content/stratified-multistage-computerized-adaptive-testing-b-blocking01945nas a2200169 4500008004100000245011100041210007100152300001200223490000700235520133200242100001301574700001601587700001201603700001001615700001601625856013401641 2001 eng d00aA comparative study of on line pretest item—Calibration/scaling methods in computerized adaptive testing0 acomparative study of on line pretest item—Calibrationscaling met a191-2120 v383 aThe purpose of this study was to compare and evaluate five on-line pretest item-calibration/scaling methods in computerized adaptive testing (CAT): marginal maximum likelihood estimate with one EM cycle (OEM), marginal maximum likelihood estimate with multiple EM cycles (MEM), Stocking's Method A, Stocking's Method B, and BILOG/Prior. The five methods were evaluated in terms ofitem-parameter recovery, using three different sample sizes (300, 1000 and 3000). The MEM method appeared to be the best choice among these, because it produced the smallest parameter-estimation errors for all sample size conditions. MEM and OEM are mathematically similar, although the OEM method produced larger errors. MEM also was preferable to OEM, unless the amount of timeinvolved in iterative computation is a concern. Stocking's Method B also worked very well, but it required anchor items that either would increase test lengths or require larger sample sizes depending on test administration design. Until more appropriate ways of handling sparse data are devised, the BILOG/Prior method may not be a reasonable choice for small sample sizes. Stocking's Method A hadthe largest weighted total error, as well as a theoretical weakness (i.e., treating estimated ability as true ability); thus, there appeared to be little reason to use it1 aBan, J C1 aHanson, B A1 aWang, T1 aYi, Q1 aHarris, D J uhttp://www.iacat.org/content/comparative-study-line-pretest-item%E2%80%94calibrationscaling-methods-computerized-adaptive-testing00492nas a2200133 4500008004100000245007900041210006900120260001500189100001000204700001400214700001600228700001600244856009800260 2001 eng d00aComparison of the SPRT and CMT procedures in computerized adaptive testing0 aComparison of the SPRT and CMT procedures in computerized adapti aSeattle WA1 aYi, Q1 aHanson, B1 aWidiatmo, H1 aHarris, D J uhttp://www.iacat.org/content/comparison-sprt-and-cmt-procedures-computerized-adaptive-testing00480nas a2200133 4500008004100000245007400041210006900115260001200184100001100196700001600207700001000223700001400233856009900247 2001 eng d00aData sparseness and online pretest calibration/scaling methods in CAT0 aData sparseness and online pretest calibrationscaling methods in aSeattle1 aBan, J1 aHanson, B A1 aYi, Q1 aHarris, D uhttp://www.iacat.org/content/data-sparseness-and-online-pretest-calibrationscaling-methods-cat00426nas a2200121 4500008004100000245005900041210005900100260001500159100001700174700001900191700001200210856008200222 2001 eng d00aDeriving a stopping rule for sequential adaptive tests0 aDeriving a stopping rule for sequential adaptive tests aSeattle WA1 aGrabovsky, I1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/deriving-stopping-rule-sequential-adaptive-tests00441nas a2200133 4500008004100000245005900041210005900100260001500159100001000174700001600184700001300200700001600213856007800229 2001 eng d00aImpact of scoring options for not reached items in CAT0 aImpact of scoring options for not reached items in CAT aSeattle WA1 aYi, Q1 aWidiatmo, H1 aBan, J-C1 aHarris, D J uhttp://www.iacat.org/content/impact-scoring-options-not-reached-items-cat00497nas a2200145 4500008004100000245007200041210006900113300001000182490000700192100001400199700001400213700001900227700001200246856009300258 2001 eng d00aOn maximizing item information and matching difficulty with ability0 amaximizing item information and matching difficulty with ability a69-770 v661 aBickel, P1 aBuyske, S1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/maximizing-item-information-and-matching-difficulty-ability00475nas a2200133 4500008004100000245007200041210006900113260001900182100001300201700001200214700001000226700001600236856008900252 2000 eng d00aEffects of nonequivalence of item pools on ability estimates in CAT0 aEffects of nonequivalence of item pools on ability estimates in  aNew Orleans LA1 aBan, J C1 aWang, T1 aYi, Q1 aHarris, D J uhttp://www.iacat.org/content/effects-nonequivalence-item-pools-ability-estimates-cat00414nas a2200109 4500008004100000245006100041210006100102260002700163100001000190700001900200856008500219 2000 eng d00aMultiple stratification CAT designs with content control0 aMultiple stratification CAT designs with content control aUnpublished manuscript1 aYi, Q1 aChang, Hua-Hua uhttp://www.iacat.org/content/multiple-stratification-cat-designs-content-control00506nas a2200145 4500008004100000245007100041210006900112260002100181100001200202700001000214700001300224700001600237700001600253856009100269 1999 eng d00aAdjusting "scores" from a CAT following successful item challenges0 aAdjusting scores from a CAT following successful item challenges aMontreal, Canada1 aWang, T1 aYi, Q1 aBan, J C1 aHarris, D J1 aHanson, B A uhttp://www.iacat.org/content/adjusting-scores-cat-following-successful-item-challenges00417nas a2200121 4500008004100000245005800041210005700099300001400156490000700170100001900177700001200196856008700208 1999 eng d00aa-stratified multistage computerized adaptive testing0 aastratified multistage computerized adaptive testing a211–2220 v231 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/stratified-multistage-computerized-adaptive-testing-001734nas a2200145 4500008004100000245005800041210005700099300001200156490000700168520126300175653003401438100001901472700001201491856008501503 1999 eng d00aa-stratified multistage computerized adaptive testing0 aastratified multistage computerized adaptive testing a211-2220 v233 aFor computerized adaptive tests (CAT) based on the three-parameter logistic mode it was found that administering items with low discrimination parameter (a) values early in the test and administering those with high a values later was advantageous; the skewness of item exposure distributions was reduced while efficiency was maintain in trait level estimation. Thus, a new multistage adaptive testing approach is proposed that factors a into the item selection process. In this approach, the items in the item bank are stratified into a number of levels based on their a values. The early stages of a test use items with lower as and later stages use items with higher as. At each stage, items are selected according to an optimization criterion from the corresponding level. Simulation studies were performed to compare a-stratified CATs with CATs based on the Sympson-Hetter method for controlling item exposure. Results indicated that this new strategy led to tests that were well-balanced, with respect to item exposure, and efficient. The a-stratified CATs achieved a lower average exposure rate than CATs based on Bayesian or information-based item selection and the Sympson-Hetter method. (PsycINFO Database Record (c) 2003 APA, all rights reserved).10acomputerized adaptive testing1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/stratified-multistage-computerized-adaptive-testing00415nas a2200121 4500008004100000245005800041210005700099300001200156490000700168100001900175700001200194856008700206 1999 eng d00aa-stratified multistage computerized adaptive testing0 aastratified multistage computerized adaptive testing a211-2220 v231 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/stratified-multistage-computerized-adaptive-testing-100525nas a2200121 4500008004100000245012600041210006900167260001600236100001100252700001200263700001000275856011800285 1999 eng d00aComparison of the a-stratified method, the Sympson-Hetter method, and the matched difficulty method in CAT administration0 aComparison of the astratified method the SympsonHetter method an aLawrence KS1 aBan, J1 aWang, T1 aYi, Q uhttp://www.iacat.org/content/comparison-stratified-method-sympson-hetter-method-and-matched-difficulty-method-cat00524nas a2200133 4500008004100000245009300041210006900134260002100203100001600224700001400240700001400254700001400268856010800282 1999 eng d00aPerformance of the Sympson-Hetter exposure control algorithm with a polytomous item bank0 aPerformance of the SympsonHetter exposure control algorithm with aMontreal, Canada1 aPastor, D A1 aChiang, C1 aDodd, B G1 aYockey, R uhttp://www.iacat.org/content/performance-sympson-hetter-exposure-control-algorithm-polytomous-item-bank00348nas a2200121 4500008004100000245003300041210003300074260001700107100001100124700001300135700001600148856006200164 1998 eng d00aAdaptive testing without IRT0 aAdaptive testing without IRT aSan Diego CA1 aYan, D1 aLewis, C1 aStocking, M uhttp://www.iacat.org/content/adaptive-testing-without-irt00427nas a2200109 4500008004100000245007600041210006900117260001700186100000700203700001400210856009300224 1998 eng d00aThe impact of nonmodel-fitting responses in a realistic CAT environment0 aimpact of nonmodelfitting responses in a realistic CAT environme aSan Diego CA1 aYi1 aNering, M uhttp://www.iacat.org/content/impact-nonmodel-fitting-responses-realistic-cat-environment00466nas a2200109 4500008004100000245009200041210006900133260001700202100000700219700001400226856011600240 1998 eng d00aNonmodel-fitting responses and robust ability estimation in a realistic CAT environment0 aNonmodelfitting responses and robust ability estimation in a rea aSan Diego CA1 aYi1 aNering, M uhttp://www.iacat.org/content/nonmodel-fitting-responses-and-robust-ability-estimation-realistic-cat-environment00559nas a2200109 4500008004100000245008700041210006900128260011900197100001000316700001400326856010900340 1998 eng d00aSimulating nonmodel-fitting responses in a CAT Environment (Research Report 98-10)0 aSimulating nonmodelfitting responses in a CAT Environment Resear aIowa City IA: ACT Inc. (Also presented at National Council on Measurement in Education, 1999: ERIC No. ED 427 042)1 aYi, Q1 aNering, L uhttp://www.iacat.org/content/simulating-nonmodel-fitting-responses-cat-environment-research-report-98-1000322nas a2200097 4500008004100000245004300041210004200084100001900126700001200145856006700157 1997 eng d00aMulti-stage CAT with stratified design0 aMultistage CAT with stratified design1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/multi-stage-cat-stratified-design00487nas a2200097 4500008004100000245012300041210006900164100001900233700001200252856012500264 1997 eng d00aNonlinear sequential designs for logistic item response theory models with applications to computerized adaptive tests0 aNonlinear sequential designs for logistic item response theory m1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/nonlinear-sequential-designs-logistic-item-response-theory-models-applications-computerized00445nas a2200109 4500008004100000245007200041210006900113260002700182100001900209700001200228856009500240 1996 eng d00aBuilding a statistical foundation for computerized adaptive testing0 aBuilding a statistical foundation for computerized adaptive test aBanff, Alberta, Canada1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/building-statistical-foundation-computerized-adaptive-testing00446nas a2200145 4500008004100000245005100041210005000092300001200142490000700154100001400161700001400175700001400189700001700203856008000220 1996 eng d00aConducting self-adapted testing using MicroCAT0 aConducting selfadapted testing using MicroCAT a821-8270 v561 aRoos, L L1 aWise, S L1 aYoes, M E1 aRocklin, T R uhttp://www.iacat.org/content/conducting-self-adapted-testing-using-microcat01733nas a2200145 4500008004100000020001400041245006700055210006500122300001200187490000700199520125900206100001901465700001201484856009101496 1996 eng d a0146-621600aA global information approach to computerized adaptive testing0 aglobal information approach to computerized adaptive testing a213-2290 v203 abased on Fisher information (or item information). At each stage, an item is selected to maximize the Fisher information at the currently estimated trait level (&thetas;). However, this application of Fisher information could be much less efficient than assumed if the estimators are not close to the true &thetas;, especially at early stages of an adaptive test when the test length (number of items) is too short to provide an accurate estimate for true &thetas;. It is argued here that selection procedures based on global information should be used, at least at early stages of a test when &thetas; estimates are not likely to be close to the true &thetas;. For this purpose, an item selection procedure based on average global information is proposed. Results from pilot simulation studies comparing the usual maximum item information item selection with the proposed global information approach are reported, indicating that the new method leads to improvement in terms of bias and mean squared error reduction under many circumstances. Index terms: computerized adaptive testing, Fisher information, global information, information surface, item information, item response theory, Kullback-Leibler information, local information, test information.1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/global-information-approach-computerized-adaptive-testing00439nas a2200121 4500008004500000245006700045210006500112300001200177490000700189100001600196700001200212856009300224 1996 Engldsh 00aA Global Information Approach to Computerized Adaptive Testing0 aGlobal Information Approach to Computerized Adaptive Testing a213-2290 v201 aChang, H -H1 aYing, Z uhttp://www.iacat.org/content/global-information-approach-computerized-adaptive-testing-100520nas a2200109 4500008004100000245009800041210006900139260004600208100001900254700001200273856012500285 1996 eng d00aRecursive maximum likelihood estimation, sequential design, and computerized adaptive testing0 aRecursive maximum likelihood estimation sequential design and co aPrinceton NJ: Educational Testing Service1 aChang, Hua-Hua1 aYing, Z uhttp://www.iacat.org/content/recursive-maximum-likelihood-estimation-sequential-design-and-computerized-adaptive-testing00518nas a2200109 4500008004100000245009900041210006900140260004200209100001200251700001900263856012600282 1995 eng d00aRecursive maximum likelihood estimation, sequential designs, and computerized adaptive testing0 aRecursive maximum likelihood estimation sequential designs and c aUniversity of Twente, the Netherlands1 aYing, Z1 aChang, Hua-Hua uhttp://www.iacat.org/content/recursive-maximum-likelihood-estimation-sequential-designs-and-computerized-adaptive-testing00387nas a2200097 4500008004100000245006200041210006200103260002900165100001000194856008500204 1985 eng d00aMethods of selecting successive items in adaptive testing0 aMethods of selecting successive items in adaptive testing aUniversity of Pittsburgh1 aYu, L uhttp://www.iacat.org/content/methods-selecting-successive-items-adaptive-testing