01626nas a2200145 4500008003900000245008600039210006900125300001200194490000700206520117400213100001701387700001801404700001301422856004501435 2019 d00aApplication of Dimension Reduction to CAT Item Selection Under the Bifactor Model0 aApplication of Dimension Reduction to CAT Item Selection Under t a419-4340 v433 aMultidimensional computerized adaptive testing (MCAT) based on the bifactor model is suitable for tests with multidimensional bifactor measurement structures. Several item selection methods that proved to be more advantageous than the maximum Fisher information method are not practical for bifactor MCAT due to time-consuming computations resulting from high dimensionality. To make them applicable in bifactor MCAT, dimension reduction is applied to four item selection methods, which are the posterior-weighted Fisher D-optimality (PDO) and three non-Fisher information-based methods—posterior expected Kullback–Leibler information (PKL), continuous entropy (CE), and mutual information (MI). They were compared with the Bayesian D-optimality (BDO) method in terms of estimation precision. When both the general and group factors are the measurement objectives, BDO, PDO, CE, and MI perform equally well and better than PKL. When the group factors represent nuisance dimensions, MI and CE perform the best in estimating the general factor, followed by the BDO, PDO, and PKL. How the bifactor pattern and test length affect estimation accuracy was also discussed.1 aMao, Xiuzhen1 aZhang, Jiahui1 aXin, Tao uhttps://doi.org/10.1177/014662161881308601066nas a2200121 4500008003900000245006100039210005900100300001200159490000700171520070300178100001800881856004500899 2018 d00aA Hybrid Strategy to Construct Multistage Adaptive Tests0 aHybrid Strategy to Construct Multistage Adaptive Tests a630-6430 v423 aHow to effectively construct multistage adaptive test (MST) panels is a topic that has spurred recent advances. The most commonly used approaches for MST assembly use one of two strategies: bottom-up and top-down. The bottom-up approach splits the whole test into several modules, and each module is built first, then all modules are compiled to obtain the whole test, while the top-down approach follows the opposite direction. Both methods have their pros and cons, and sometimes neither is convenient for practitioners. This study provides an innovative hybrid strategy to build optimal MST panels efficiently most of the time. Empirical data and results by using this strategy will be provided.1 aXiong, Xinhui uhttps://doi.org/10.1177/014662161876273901532nas a2200169 4500008003900000245007100039210006900110300001200179490000700191520103000198100001601228700001801244700001801262700001801280700001901298856004501317 2018 d00aLatent Class Analysis of Recurrent Events in Problem-Solving Items0 aLatent Class Analysis of Recurrent Events in ProblemSolving Item a478-4980 v423 aComputer-based assessment of complex problem-solving abilities is becoming more and more popular. In such an assessment, the entire problem-solving process of an examinee is recorded, providing detailed information about the individual, such as behavioral patterns, speed, and learning trajectory. The problem-solving processes are recorded in a computer log file which is a time-stamped documentation of events related to task completion. As opposed to cross-sectional response data from traditional tests, process data in log files are massive and irregularly structured, calling for effective exploratory data analysis methods. Motivated by a specific complex problem-solving item “Climate Control” in the 2012 Programme for International Student Assessment, the authors propose a latent class analysis approach to analyzing the events occurred in the problem-solving processes. The exploratory latent class analysis yields meaningful latent classes. Simulation studies are conducted to evaluate the proposed approach.1 aXu, Haochen1 aFang, Guanhua1 aChen, Yunxiao1 aLiu, Jingchen1 aYing, Zhiliang uhttps://doi.org/10.1177/014662161774832501393nas a2200169 4500008004100000245003600041210003600077260005500113520088800168653000801056653002101064100002101085700001201106700001601118700001801134856007101152 2017 eng d00aItem Pool Design and Evaluation0 aItem Pool Design and Evaluation aNiigata, JapanbNiigata Seiryo Universityc08/20173 a

Early work on CAT tended to use existing sets of items which came from fixed length test forms. These sets of items were selected to meet much different requirements than are needed for a CAT; decision making or covering a content domain. However, there was also some early work that suggested having items equally distributed over the range of proficiency that was of interest or concentrated at a decision point. There was also some work that showed that there was bias in proficiency estimates when an item pool was too easy or too hard. These early findings eventually led to work on item pool design and, more recently, on item pool evaluation. This presentation gives a brief overview of these topics to give some context for the following presentations in this symposium.

Session Video

10aCAT10aItem Pool Design1 aReckase, Mark, D1 aHe, Wei1 aXu, Jing-Ru1 aZhou, Xuechun uhttps://drive.google.com/open?id=1ZAsqm1yNZlliqxEHcyyqQ_vOSu20xxZs01415nas a2200169 4500008003900000245007000039210006900109300001400178490000700192520091000199100001901109700002101128700001601149700001201165700001401177856005401191 2015 d00aInvestigation of Response Changes in the GRE Revised General Test0 aInvestigation of Response Changes in the GRE Revised General Tes a1002-10200 v753 aResearch on examinees’ response changes on multiple-choice tests over the past 80 years has yielded some consistent findings, including that most examinees make score gains by changing answers. This study expands the research on response changes by focusing on a high-stakes admissions test—the Verbal Reasoning and Quantitative Reasoning measures of the GRE revised General Test. We analyzed data from 8,538 examinees for Quantitative and 9,140 for Verbal sections who took the GRE revised General Test in 12 countries. The analyses yielded findings consistent with prior research. In addition, as examinees’ ability increases, the benefit of response changing increases. The study yielded significant implications for both test agencies and test takers. Computer adaptive tests often do not allow the test takers to review and revise. Findings from this study confirm the benefit of such features.1 aLiu, Ou, Lydia1 aBridgeman, Brent1 aGu, Lixiong1 aXu, Jun1 aKong, Nan uhttp://epm.sagepub.com/content/75/6/1002.abstract01583nas a2200133 4500008003900000245012700039210006900166300001200235490000700247520111200254100001701366700001301383856005301396 2013 d00aThe Application of the Monte Carlo Approach to Cognitive Diagnostic Computerized Adaptive Testing With Content Constraints0 aApplication of the Monte Carlo Approach to Cognitive Diagnostic  a482-4960 v373 a

The Monte Carlo approach which has previously been implemented in traditional computerized adaptive testing (CAT) is applied here to cognitive diagnostic CAT to test the ability of this approach to address multiple content constraints. The performance of the Monte Carlo approach is compared with the performance of the modified maximum global discrimination index (MMGDI) method on simulations in which the only content constraint is on the number of items that measure each attribute. The results of the two simulation experiments show that (a) the Monte Carlo method fulfills all the test requirements and produces satisfactory measurement precision and item exposure results and (b) the Monte Carlo method outperforms the MMGDI method when the Monte Carlo method applies either the posterior-weighted Kullback–Leibler algorithm or the hybrid Kullback–Leibler information as the item selection index. Overall, the recovery rate of the knowledge states, the distribution of the item exposure, and the utilization rate of the item bank are improved when the Monte Carlo method is used.

1 aMao, Xiuzhen1 aXin, Tao uhttp://apm.sagepub.com/content/37/6/482.abstract01189nas a2200133 4500008003900000245003700039210003700076300001200113490000700125520082900132100001900961700001800980856005700998 2013 d00aSpeededness and Adaptive Testing0 aSpeededness and Adaptive Testing a418-4380 v383 a

Two simple constraints on the item parameters in a response–time model are proposed to control the speededness of an adaptive test. As the constraints are additive, they can easily be included in the constraint set for a shadow-test approach (STA) to adaptive testing. Alternatively, a simple heuristic is presented to control speededness in plain adaptive testing without any constraints. Both types of control are easy to implement and do not require any other real-time parameter estimation during the test than the regular update of the test taker’s ability estimate. Evaluation of the two approaches using simulated adaptive testing showed that the STA was especially effective. It guaranteed testing times that differed less than 10 seconds from a reference test across a variety of conditions.

1 aLinden, Wim, J1 aXiong, Xinhui uhttp://jeb.sagepub.com/cgi/content/abstract/38/4/41800429nas a2200121 4500008003900000245006500039210006500104300001200169490000700181100001000188700001500198856009400213 2006 d00aComputerized adaptive testing under nonparametric IRT models0 aComputerized adaptive testing under nonparametric IRT models a121-1370 v711 aXu, X1 aDouglas, J uhttp://www.iacat.org/content/computerized-adaptive-testing-under-nonparametric-irt-models02433nas a2200241 4500008004100000020004600041245008600087210006900173260002500242300001200267490000700279520159900286653001801885653002401903653002001927653002801947653002101975653004001996653001402036100001802050700001202068856011102080 2006 eng d a0895-7347 (Print); 1532-4818 (Electronic)00aOptimal and nonoptimal computer-based test designs for making pass-fail decisions0 aOptimal and nonoptimal computerbased test designs for making pas bLawrence Erlbaum: US a221-2390 v193 aNow that many credentialing exams are being routinely administered by computer, new computer-based test designs, along with item response theory models, are being aggressively researched to identify specific designs that can increase the decision consistency and accuracy of pass-fail decisions. The purpose of this study was to investigate the impact of optimal and nonoptimal multistage test (MST) designs, linear parallel-form test designs (LPFT), and computer adaptive test (CAT) designs on the decision consistency and accuracy of pass-fail decisions. Realistic testing situations matching those of one of the large credentialing agencies were simulated to increase the generalizability of the findings. The conclusions were clear: (a) With the LPFTs, matching test information functions (TIFs) to the mean of the proficiency distribution produced slightly better results than matching them to the passing score; (b) all of the test designs worked better than test construction using random selection of items, subject to content constraints only; (c) CAT performed better than the other test designs; and (d) if matching a TIP to the passing score, the MST design produced a bit better results than the LPFT design. If an argument for the MST design is to be made, it can be made on the basis of slight improvements over the LPFT design and better expected item bank utilization, candidate preference, and the potential for improved diagnostic feedback, compared with the feedback that is possible with fixed linear test forms. (PsycINFO Database Record (c) 2007 APA, all rights reserved)10aadaptive test10acredentialing exams10aDecision Making10aEducational Measurement10amultistage tests10aoptimal computer-based test designs10atest form1 aHambleton, RK1 aXing, D uhttp://www.iacat.org/content/optimal-and-nonoptimal-computer-based-test-designs-making-pass-fail-decisions00528nas a2200109 4500008004100000245009800041210006900139260006200208100001800270700001200288856011800300 2004 eng d00aComputer-based test designs with optimal and non-optimal tests for making pass-fail decisions0 aComputerbased test designs with optimal and nonoptimal tests for aResearch Report, University of Massachusetts, Amherst, MA1 aHambleton, RK1 aXing, D uhttp://www.iacat.org/content/computer-based-test-designs-optimal-and-non-optimal-tests-making-pass-fail-decisions01470nas a2200133 4500008003900000245013600039210006900175300000900244490000700253520098400260100001601244700002501260856005101285 2004 d00aImpact of Test Design, Item Quality, and Item Bank Size on the Psychometric Properties of Computer-Based Credentialing Examinations0 aImpact of Test Design Item Quality and Item Bank Size on the Psy a5-210 v643 a

Computer-based testing by credentialing agencies has become common; however, selecting a test design is difficult because several good ones are available—parallel forms, computer adaptive (CAT), and multistage (MST). In this study, three computerbased test designs under some common examination conditions were investigated. Item bank size and item quality had a practically significant impact on decision consistency and accuracy. Even in nearly ideal situations, the choice of test design was not a factor in the results. Two conclusions follow from the findings: (a) More time and resources should be committed to expanding the size and quality of item banks, and (b) designs that individualize an exam administration such as MST and CAT may not be helpful when the primary purpose of the examination is to make pass-fail decisions and conditions are present for using parallel forms with a target information function that can be centered on the passing score.

1 aXing, Dehui1 aHambleton, Ronald, K uhttp://epm.sagepub.com/content/64/1/5.abstract00524nas a2200133 4500008004100000245009600041210006900137300001200206490000700218100001600225700001800241700001200259856011900271 2003 eng d00aPsychometric and psychological effects of item selection and review on computerized testing0 aPsychometric and psychological effects of item selection and rev a791-8080 v631 aRevuelta, J1 aXiménez, M C1 aOlea, J uhttp://www.iacat.org/content/psychometric-and-psychological-effects-item-selection-and-review-computerized-testing01785nas a2200145 4500008003900000245009600039210006900135300001200204490000700216520130200223100002101525700002401546700001601570856005301586 2003 d00aPsychometric and Psychological Effects of Item Selection and Review on Computerized Testing0 aPsychometric and Psychological Effects of Item Selection and Rev a791-8080 v633 a

Psychometric properties of computerized testing, together with anxiety and comfort of examinees, are investigated in relation to item selection routine and the opportunity for response review. Two different hypotheses involving examinee anxiety were used to design test properties: perceived control and perceived performance. The study involved three types of administration of a computerized English test for Spanish speakers (adaptive, easy adaptive, and fixed) and four review conditions (no review, review at end, review by blocks of 5 items, and review item-by-item). These were applied to a sample of 557 first-year psychology undergraduate students to examine main and interaction effects of test type and review on psychometric and psychological variables. Statistically significant effects were found in test precision among the different types of test. Response review improved ability estimates and increased testing time. No psychological effects on anxiety were found. Examinees in all review conditions considered more important the possibility of review than those who were not allowed to review. These results concur with previous findings on examinees' preference for item review and raise some issues that should be addressed in the field of tests with item review.

1 aRevuelta, Javier1 aXiménez, Carmen, M1 aOlea, Julio uhttp://epm.sagepub.com/content/63/5/791.abstract00457nas a2200121 4500008004100000245007300041210006900114260001500183100001000198700001900208700001500227856009300242 2003 eng d00aA simulation study to compare CAT strategies for cognitive diagnosis0 asimulation study to compare CAT strategies for cognitive diagnos aChicago IL1 aXu, X1 aChang, Hua-Hua1 aDouglas, J uhttp://www.iacat.org/content/simulation-study-compare-cat-strategies-cognitive-diagnosis01711nas a2200169 4500008004100000245015800041210006900199300001000268490000700278520105400285100001901339700001801358700001601376700001201392700001601404856012101420 2003 eng d00aSmall sample estimation in dichotomous item response models: Effect of priors based on judgmental information on the accuracy of item parameter estimates0 aSmall sample estimation in dichotomous item response models Effe a27-510 v273 aLarge item banks with properly calibrated test items are essential for ensuring the validity of computer-based tests. At the same time, item calibrations with small samples are desirable to minimize the amount of pretesting and limit item exposure. Bayesian estimation procedures show considerable promise with small examinee samples. The purposes of the study were (a) to examine how prior information for Bayesian item parameter estimation can be specified and (b) to investigate the relationship between sample size and the specification of prior information on the accuracy of item parameter estimates. The results of the simulation study were clear: Estimation of item response theory (IRT) model item parameters can be improved considerably. Improvements in the one-parameter model were modest; considerable improvements with the two- and three-parameter models were observed. Both the study of different forms of priors and ways to improve the judgmental data used in forming the priors appear to be promising directions for future research. 1 aSwaminathan, H1 aHambleton, RK1 aSireci, S G1 aXing, D1 aRizavi, S M uhttp://www.iacat.org/content/small-sample-estimation-dichotomous-item-response-models-effect-priors-based-judgmental00517nas a2200109 4500008004100000245012800041210006900169260001600238100001200254700001800266856012300284 2002 eng d00aImpact of test design, item quality and item bank size on the psychometric properties of computer-based credentialing exams0 aImpact of test design item quality and item bank size on the psy aNew Orleans1 aXing, D1 aHambleton, RK uhttp://www.iacat.org/content/impact-test-design-item-quality-and-item-bank-size-psychometric-properties-computer-based00621nas a2200109 4500008004100000245018300041210006900224260006700293100001200360700001800372856012100390 2001 eng d00aImpact of several computer-based testing variables on the psychometric properties of credentialing examinations (Laboratory of Psychometric and Evaluative Research Report No 393)0 aImpact of several computerbased testing variables on the psychom aAmherst, MA: University of Massachusetts, School of Education.1 aXing, D1 aHambleton, RK uhttp://www.iacat.org/content/impact-several-computer-based-testing-variables-psychometric-properties-credentialing-000500nas a2200109 4500008004100000245011600041210006900157260001500226100001200241700001800253856011900271 2001 eng d00aImpact of several computer-based testing variables on the psychometric properties of credentialing examinations0 aImpact of several computerbased testing variables on the psychom aSeattle WA1 aXing, D1 aHambleton, RK uhttp://www.iacat.org/content/impact-several-computer-based-testing-variables-psychometric-properties-credentialing00562nas a2200133 4500008004100000245012400041210006900165260001900234100001300253700001200266700001400278700001500292856012100307 2000 eng d00aClassification accuracy and test security for a computerized adaptive mastery test calibrated with different IRT models0 aClassification accuracy and test security for a computerized ada aNew Orleans LA1 aRobin, F1 aXing, D1 aScrams, D1 aPotenza, M uhttp://www.iacat.org/content/classification-accuracy-and-test-security-computerized-adaptive-mastery-test-calibrated00549nas a2200145 4500008004500000245009400045210006900139300001200208490000700220100001200227700001600239700001700255700001400272856011700286 2000 Spandsh 00aPsychometric and psychological effects of review on computerized fixed and adaptive tests0 aPsychometric and psychological effects of review on computerized a157-1730 v211 aOlea, J1 aRevuelta, J1 aXimenez, M C1 aAbad, F J uhttp://www.iacat.org/content/psychometric-and-psychological-effects-review-computerized-fixed-and-adaptive-tests00561nam a2200097 4500008004100000245016500041210006900206260005500275100001200330856012100342 1993 eng d00aComputerized adaptive testing strategies: Golden section search, dichotomous search, and Z-score strategies (Doctoral dissertation, Iowa State University, 1990)0 aComputerized adaptive testing strategies Golden section search d aDissertation Abstracts International, 54-03B, 17201 aXiao, B uhttp://www.iacat.org/content/computerized-adaptive-testing-strategies-golden-section-search-dichotomous-search-and-z00368nas a2200085 4500008004100000245006800041210006800109100001200177856009300189 1990 eng d00aDichotomous search strategies for computerized adaptive testing0 aDichotomous search strategies for computerized adaptive testing1 aXiao, B uhttp://www.iacat.org/content/dichotomous-search-strategies-computerized-adaptive-testing00403nas a2200097 4500008004100000245007100041210006900112260001600181100001200197856009600209 1989 eng d00aGolden section search strategies for computerized adaptive testing0 aGolden section search strategies for computerized adaptive testi aBerkeley CA1 aXiao, B uhttp://www.iacat.org/content/golden-section-search-strategies-computerized-adaptive-testing