Table of contents

Computer-Based Testing (Yves A. Devriendt)

Abstract

Ever since the development of the first modern test a century ago, psychological testing has been influenced thoroughly by changes in technology. Now, a century later, the computer has become a central tool in test construction and administration: New theoretical insights about the conceptualization of tests have resulted in innovative assessment instruments and more people now can be tested better in a shorter time. Furthermore it is shown in this reading that Computer–Based Testing systems can be developed in different ways. Despite all the advantages Computer–Based Testing remains costly and requires specialised knowledge. Nevertheless, the conclusion is that advantages outnumber disadvantages and it is expected that Computer–Based Testing and new technological changes will continue to impact upon future test developments.

Main text

Click here for the full text (pdf). It will open in a new page.

Reference list

Bennet, R.E. (Februari 2001). How the Internet Will Help Large–Scale Assessment Reinvent Itself. Educational Policy Analysis Archives, 9, 5. Retrieved February 23, 2005, from http://www.epaa.asu.edu/epaa/v9n5.html

Crocker, L.L., & Algina, J. (1986).Introduction to Classical and Modern Testing. New York: Holt, Rinehart & Winston.

Drasgow, F., & Olson–Buchanan, J.B. (1999). Innovations in Computerized Assessment. Mahweh, NJ: Lawrence Erlbaum Associates.

Hardinge, N.M. (1997). Selection of Military Staff. In N. Anderson, & P. Herriot, International Handbook of Selection and Assessment (pp. 161–181). Chichester: John Wiley and Sons.

Irvine, S.H., & Kyllonen, P. (2002). Item Generation for Test Development. Hillsdale, NJ: Lawrence Erlbaum Associates.

International Test Commission (ITC). ( 2005). International Guidelines on Computer–Based and Internet–Delivered Testing (Version 1.0, January 2005). Retrieved February 23, 2005, from http://www.intest.org/

Jansen, P.G.W. (1997). Assessment in a Technological World. In N. Anderson, & P. Herriot, International Handbook of Selection and Assessment (pp. 125–145). Chichester: John Wiley and Sons.

Lievens, F., & Coetsier, P. (2002). Situational Tests in Student Selection: An Examination of Predictive Validity, Adverse Impact, and Construct Validity. International Journal of Selection and Assessment, 10, 245-257.

McBride, J.R. (1998). Innovations in Computer–Based Ability Testing: Promise, Problems, and Perils. In M.D. Hakel, Multiple Choice: Evaluating Alternatives to Traditional Testing for Selection (pp. 23–39). Mahwah, New Jersey: Lawrence Erlbaum Associates.

Neuman, G., & Baydoun, R (1998). Computerization of pencil and paper tests: When are they equivalent? Applied Psychological Measurement, 22, 71–83.

McIntire, S.A., & Miller, L.A. (2000). Foundations of Psychological Testing. Boston: McGraw–Hill Companies, Inc.

Parshall, C.G., Spray, J.A., Kalohn, J.C., & Davey, T. (2002). Practical Considerations in Computer–Based Testing. NY: Springer.

Russel, T.L., & Peterson, N.G. (1997).The Test Plan. In D.L. Whetzel, & G.R. Wheaton, Applied Measurement Methods in Industrial psychology (pp. 115–139). Palo Alto, CA: Consulting Psychologists Press.

Schultz, D.P., & Schultz, S.E. (1992). A History of Modern Psychology. New York: Harcourt Brace Jovanovich, Inc.

Steege, F.W., & Fritscher, W. (1991). Psychological Assessment and Personnel Management. In R. Gal & A.D. Mangelsdorff, Handbook of Military Psychology (pp. 7–36). Chichester: John Wiley and Sons.

Sturman, L., & Kispal, A. (2003). To e or not to e? A comparison of electronic marking and paper-based marking. Paper presented at the 29th International Association for Educational Assessment Conference, 5-10 October 2003, Manchester, UK.

Wainer, H., Dorans, N.J., Eignor, D., Flaugher; R., Green, B.F, Mislevy. R.J., Steinberg, L., & Thissen, D., (2000). Computerized Adaptive Testing: a Primer. Mahwah, NJ: Lawrence Erlbaum Associates.

Weiss, D.J. (1983). New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing. New York: Academic Press.

Zakrewski, S., & Bull; J (1998). The Mass Implementation and Evaluation of Computer–Based Assessments. Assessment and Evaluation in Higher Education, 23, 141–152.

Questions/Items for classroom discussion

1 a What is meant by p&p tests? Provide two examples.

1 b What is meant by CBT? Provide two examples.

1 c What do you think the advantages and disadvantages are of p&p tests and CBT?

2 a Give an overview of test evolvement beginning with modern test theory. Sum up the key test concepts or test periods.

2 b Compare each test concept using degree of automation as a criterion. Talk about the differences between each test concept in a group discussion.

2 c In which different ways can computers be used in test implementation?

3 Why are many potential users still reluctant to implement CBT?

4 Binet conceived the first intelligence test in 1905. Is this test currently still of importance? Explain your answer.

5 Is CBT always to be preferred to p&p test procedures? Explain your answer.

6 a Determine the steps in traditional test development (use this reading, another reading or a handbook).

6 b Determine the steps to take to construct a CBT system.

6 c What are the similarities and dissimilarities between traditional test development and CBT?

6 d Discuss 6a, 6b and 6c within your group members.

7 a Explain the different stages that McBride distinguishes in the evolution of CBT.

7 b Are there other classifications or other criteria possible? Use any available literature, documentation or other readings.

8 a In which domains can a test user implement CBT? Give an example from the reading and explain your answer. Give an example from outside of the reading and explain your answer.

8 b Which criteria are used to determine whether it is practical and cost effective to implement CBT?

9 a Sum up and explain possible future developments concerning CBT.

9 b Do you agree with the predicted developments mentioned in the text?

9 c Think of other possible future developments and discuss with your group members.

Case study “Test Company” (TC)

Organisation TC, which tests about 10 000 students a year, has equipped its administrative departments with computers. The organization would like to computerise its test procedures, but has already used all the finances to improve the administrative departments. As a consequence, the organization continues using existing tests, which use OCR to transform answers to test scores. A few years later, the management of TC copies the currently used tests on the computer and intends to investigate the equivalence of the computerised tests to the p&p tests. Because these tests are speed tests there appears to be little or no equivalence. So, TC management decides to buy an IGT system.

10 a Which generations of problems do you recognise in this story? Justify your answer by referring to the text of this case study.

10 b Discuss your opinion with your group members. Case study “Hospital Diagnostics” (HD)

HD medical doctors routinely treat patients with a specific brain injury. They have a problem, as relatively few clinical (neuro)psychological tests are available on the market to test patients with this type of brain injury. In fact, there is only one effective test, but it takes a long time to fill out, and as a consequence patients are extremely tired and stressed when they have finished the test. HD management considers using CBT, but is afraid of the costs involved with test development.

11 a What are the possible solutions for developing and implementing CBT? Give your answer in detail.

11 b Which procedure do you suggest exactly? Why?

11 c Give an overview of all the arguments that played a role in your proposal.

11 d Discuss the different solutions with your group members.

Biographical statement

Yves Devriendt is a psychologist who graduated from Ghent University (Belgium). He obtained a teaching certificate from Ghent University and a master’s degree in “ Quantitative Analysis in the Social Sciences” from the Brussels Catholic University. In his career with the Belgian Department of Defence as a military psychologist, he has twelve years of experience as an interviewer, assessor and trainer within a military context. Since 1996 he has been working as a researcher and test developer for the Belgian Department of Defence. Recently he worked on projects about automated test batteries, cognitive tests, selection procedures and retention. He publishes and speaks on topics such as computer-based testing, test batteries, test validation, military selection, retention and the position of the psychologist in society.

Address correspondence to:

Y.A. Devriendt, Lic Psy, MSc Quantitative Analysis
Belgian Department of Defence
Directorate General of Human Resources
Accession Policy – Research & Development
Kwartier Koningin Astrid
Bruynstraat 1
1120 Brussels (Neder–over–Heembeek)
BELGIUM
Email: yves.devriendt@mil.be