home > guidelines > computer based testing International Test Commission

Give due regard to technological issues in Computer-based (CBT) and Internet Testing

Give consideration to hardware and software requirements

  • Verify the documented minimum hardware, software or browser requirements to ensure that they are communicated clearly to the user. Ensure that other technical and operational requirements for the test are explained to the user.
  • Confirm that adequate testing of the system has been completed and documented on the appropriate delivery platforms stated to be suitable.
  • Use only software or hardware features that are essential for measuring the construct and that are likely to be available on systems used by the intended test users and test-takers.
  • Ensure that the test will be as easy as possible to support and maintain in light of likely developments in hardware and software (operating systems etc).
  • Test and document any new features added to the program after publication.

    •  

    Take account of the robustness of the CBT/Internet test

  • Confirm the robustness of the system has been checked across a range of suitable platforms.
  • Provide sufficient redundancy on all systems throughout the testing site (including incoming and outgoing communications) to allow the site to operate even if one of its components fails.
  • Check the degree to which the test prevents user errors from causing administration problems. Provide users with guidance on what to do in the event that ‘bugs’ occur during testing (e.g. a test user should be able to report bugs and problems that may be experienced during the testing process).
  • Provide users with contact details (e.g., telephone number, internet address) for technical support.
  • Confirm that the CBT/Internet test responds in a timely manner when taking the test. Where this does not occur, inform test developers and discontinue use of the test until the problem is solved.
  • For Internet testing, put procedures in place to deal fairly with the impact of hang-ups, lost connections and slow downloads. Where download or other technical problems occur, advise the test user/taker of alternatives (e.g., using alternative media or an alternative venue).
  • Document and disseminate relevant technical support to test users. Where appropriate, offer technical support services with trained staff.

    •  

    Consider human factors issues in the presentation of material via computer or the Internet

  • Verify that screen design issues have been taken into account in the development of the CBT/Internet test. Where problems are noticed, provide clear and detailed information about the problems to the test developer.
  • Verify that item presentation is consistent throughout the test.
  • Verify that appropriate and informative error messages are presented when necessary.

    •  

    Consider reasonable adjustments to the technical features of the test for candidates with disabilities

  • Confirm that the hardware/software features of the CBT/Internet test facilitate the participation of test-takers with disabilities and those with special needs (e.g., those who need larger page font).
  • Inform test users about the types of accommodations and modifications that can be made for test-takers with disabilities and those with special needs.
  • Inform test users of the acceptable limits to which tests can be modified or accommodations provided to test-takers.
  • Ensure that test modification and accommodations provided to test users are consistent with legislation regarding individuals with disabilities and special needs.

    •  

    Provide help , information, and practice items within the CBT/Internet test

  • Provide technical support documentation at a level appropriate for test users. Where appropriate, provide additional customer support services.
  • Disseminate instructions on how to set-up the system to test users. For Internet testing, inform, where appropriate, test users on how to log a test-taker on and off the system.
  • Provide clear and sufficient on-screen instructions.
  • Where appropriate, verify that suitable practice items and tutorials are available. For Internet testing , provide procedures to verify whether a test-taker has accessed practice items and tutorials. Often a test cannot be started until certain practice items have been completed.

    • Attend to quality issues in CBT and Internet testing

      Ensure knowledge, competence and appropriate use of CBT/Internet testing

  • Ensure that the CBT/Internet test is suitable in terms of content and technical adequacy for its stated purpose and intended test-taker groups.
  • Provide test users with sufficient information about the CBT/Internet test, its modes of operation, and basic computer functions. If appropriate, provide training materials that are specific to CBT/Internet tests and testing.
  • Provide test users with ‘best practice’ testing policies.
  • Provide test users with clear instructions on how to correctly access and administer Internet tests, including how to log test-takers onto the system.
  • Maintain and regularly update documentation relating to CBT/Internet testing, including pertinent changes in legislation and policy.
  • Adhere to legal, professional, and ethical mandates related to CBT/Internet testing.
  • For Internet testing, document the limitations of the test in terms of the professional context in which it operates:
    • provide a statement indicating the limitations of the relationships between test user and test-taker that can be achieved through this mode (e.g. the Internet is a impersonal medium and a test user may provide only limited advice)
    • provide a statement stating that there are limitations to the conclusions that can be reached just using the Internet test scores.

    •  

    Consider the psychometric qualities of the CBT/Internet test

  • Provide appropriate documentation for the psychometric properties of the CBT/Internet test.
  • Ensure that current psychometric standards (test reliability, validity etc.) have been met even though the way in which the tests are developed and delivered may differ.
  • Publish and offer online only those tests that have appropriate psychometric evidence to support their use.
  • When offering assessments online, give advice to test users as to what to look for in order to help them distinguish between tests with and without documented psychometric properties.
  • Verify that the CBT/Internet test does not require knowledge, skills or abilities that are irrelevant to the construct being assessed.
  • Provide documentation that describes the algorithms and measurement models used and present evidence showing that the test has been validated using these algorithms or models.
  • For tests based on models that may be unfamiliar to test users, provide explanations of the relevant concepts for the user.
  • Verify that psychometric model fit has been re-evaluated when changes are made to the test content.

    •  

    Where the CBT/Internet test has been developed from a paper and pencil version, ensure that there is evidence of equivalence

  • Evaluate the documented evidence of the equivalence of the CBT/Internet test, especially if norms from manual versions are to be used by test users to interpret scores on a computerised version of the test.
  • If the developer does not provide evidence of equivalence (e.g., comparable reliabilities, etc.), conduct appropriate equivalence studies.
  • If the developer does not provide evidence relating to the use of the test under conditions that represent those that the intended target population will experience (e.g., unproctored, unstandardised testing), additional studies of test equivalence and norming should be conducted.
  • Verify that the technical features of the CBT/Internet test (e.g., test-taker control and item presentation) allow the results from the CBT/Internet test to be equivalent to the manual version.

    •  

    Score and analyse CBT/Internet testing results accurately

  • Confirm that the accuracy of scoring rules has been adequately evaluated prior to test use.
  • Inform test users about the scoring rules employed within the CBT/Internet test (e.g., use of non-scored items, penalties for guessing).
  • Inform test users how CBTI statements are derived and the validity of that methodology.
  • Stress to test users the importance of carefully checking data input by hand into a computer for scoring.

    •  

    Interpret results appropriately and provide appropriate feedback

  • Inform test users of the potential limitations of interpreting results using CBTI. Specifically:
    • statements in a report may be general and not directed towards the specific purpose of the assessment (or specific individuals);
    • interpretation is based only on scores of those tests whose data were used as input; therefore, other ancillary data which may be important cannot be taken into account (e.g., scores on other, non-computerised, forms of assessment);
    • for open or controlled modes of Internet testing, test-takers may have been tested in non-standardised, unproctored, or variable conditions, whereas score interpretations are based on administration in proctored, standardised conditions;
    • some tests are completed in an administration mode that makes it impossible to guarantee the true identity of the test-taker.
  • Assess the suitability of the CBTI provided within the CBT/Internet test system. In particular, take note of
    • evidence of the validity and utility of reports,
    • the coverage of the reports,
    • the consistency of the reports based on similar sets of data,
    • the acceptability of the report to intended audiences,
    • time, cost and length implications for a test user, and
    • freedom from systematic bias.
  • Advise test users on how best to share CBTI with test-takers and other relevant stakeholders.
  • Inform test users of ethical and other accepted practice issues related to providing CBTI feedback to test-takers.

    •  

    Consider equality of access for all groups

  • Where possible, encourage test users to collect biographical data on test-takers in order to monitor the number of people from protected/minority groups who take any CBT/Internet test.
  • Where unequal access to CBT/Internet tests may occur, recommend that test users make alternative forms of assessment available.
  • Inform test users of any evidence regarding DIF for different test-taker groups.
  • When tests are published internationally, provide test users with advice on how to ensure equivalent access to computer technology or the Internet for geographically-diverse groups of test-takers.
  • Where an adapted version of a test is available, provide documentation specifying the equivalence of the adaptation to the original assessment.

    • Provide appropriate levels of control over CBT and Internet testing

     Detail the level of control over the test conditions

  • Provide sufficient details to test users on hardware, software, and procedural requirements for administering the CBT/Internet test.
  • Describe the test taking conditions candidates should consider when undertaking an Internet-based test.
  • Inform test users of the need to consider health and safety rules during CBT/Internet testing. For example, identify whether an Internet test has the facility for breaks if the testing process is lengthy.

    •  

    Detail the appropriate control over the supervision of the testing

  • Document the level of supervision expected for the CBT/Internet test.
  • Specify and restrict the use of specific CBT/Internet tests for particular testing scenarios. For example, psychometric tests for use in post-sift selection testing and/or post-hire assessment normally would not be available in open mode.

    •  

    Give due consideration to controlling prior practice and item exposure

  • Verify that Internet-based maximum performance tests have appropriate controls to reduce item exposure.
  • Provide test users with sufficient information on and training in how to control item exposure.
  • Where appropriate, provide test-takers with practice without compromising the security of the test items.

    •  

    Give consideration to control over test-taker’s authenticity and cheating

  • Detail the level of authentication required to access various parts of the assessment system, based on the mode of operation used. Exercise control by requiring test users (in the Supervised and Managed modes) and test-takers (in the Controlled mode) to use a username and password when accessing the test.
  • For moderate or high stakes assessment involving multiple stages, provide information on how test users can reduce the risk of test-taker cheating (e.g., having another person to take the test as a proxy). Where an assessment is carried out in open or controlled mode, checks against cheating can be carried out by requiring the test-taker to undertake a subsequent validation assessment in proctored conditions (i.e. supervised or managed conditions) and a comparison of scores made.
  • Identify the threats to test validity that exist if test control is not maintained properly.
  • Provide advice on the design and implementation of 'honesty (honor) policies' in assessment procedures if one or more stages of the process are to be carried out without direct human supervision.

    •  

    Make appropriate provision for security and safeguarding privacy in CBT and Internet testing

     Take account of the security of test materials

  • Protect sensitive features of the test from illegitimate disclosure. For Internet testing, all important intellectual property (e.g., scoring rules, norms, interpretation algorithms) associated with a test should remain on the host server. Only test items and the outputs from report generators usually should appear on the test user’s or test-taker’s screens.
  • Where appropriate, develop a policy that limits test material access to qualified and authorised test users and testing centres. For example, when testing over the Internet, test users would need to obtain and use a password before they were able to access test materials or set up an assessment for a test-taker.
  • Passwords should be issued only to users qualified to use the Internet test.
  • Verify and check that the CBT/Internet test has features to protect it from illegal hacking and computer viruses. Confirm for Internet testing that reasonable steps have been taken to prevent servers from being accessed by unauthorised or illegal means.
  • For Internet testing, maintain control over the sensitive features of the test and report copyright violations on the Internet. M onitor the web for illegal versions, old/outdated versions and part versions of the Internet test and take steps (e.g., enforcing copyright law) to eliminate these violations.
  • Take steps to secure protection of test content under existing laws.
  • Take appropriate measures to identify stolen test material on the Internet and to estimate its impact of its distribution on the testing program.
  • Take appropriate measures to control the distribution of stolen test material on the Internet including notification of appropriate legal authorities.
  • Maintain a process for the adjudication of security breach allegations and specify appropriate sanctions.

    •  

    Consider the security of test-taker’s data transferred over the Internet

  • Maintain the security of test-taker data transmitted over the Internet (e.g. by encryption).
  • Ensure that test users and test-takers are informed that the host server has correctly received their data.
  • Inform test users of their rights and obligations in relation to local data protection and privacy legislation.
  • Conduct regular and frequent backups of all collected data and provide test users with a detailed disaster recovery plan should problems emerge.

    •  

    Maintain the confidentiality of test-taker results

  • When test data must be stored with publishers, specify the procedures and systems to maintain the confidentiality and security of data.
  • Inform test users of who has access to test data, for what purposes, and how long the data will be stored electronically.
  • Adhere to country-specific data protection laws/regulations governing the storage of personal data.
  • Restrict access to personal data stored on the host server to those who are qualified and authorised .
  • Protect all sensitive personal material held on computer, disk, or a server with robust (non-trivial) encryption devices or passwords.
  • Confirm the security and confidentiality of the backup data when used to store sensitive personal data.