Table of contents
ASSESSMENT CENTERS: OVERVIEW OF PRACTICE AND RESEARCHAbstract
ASSESSMENT CENTERS: OVERVIEW OF PRACTICE AND RESEARCH
Basic Features of Assessment Centers
Some Examples of Assessment Center Exercises
Design of Assessment Centers
Brief History of Assessment Centers
Recent Developments in Assessment Center Practice
Applying Assessment Centers in Multi-national and Cross-cultural Settings
Use of Technology and Virtual Assessment Centers
Assessment Centers for Developmental Purposes
Research Evidence on Assessment Centers
Criterion-Related Validity
Reliability
Utility
Adverse Impact
Applicant Reactions
Coachability and Fakability
Construct-related Validity
Conclusions
References
Questions for classroom discussion
Author information
ASSESSMENT CENTERS: OVERVIEW OF PRACTICE AND RESEARCH
FILIP LIEVENS & EVELINE SCHOLLAERT
Ghent University Belgium
Reviewed by René Butter, February 2010
Uploaded June 2009
Abstract
This chapter presents assessment centers (ACs) as a procedure. First, the basic characteristics of ACs are delineated. Second, some examples of common AC exercises are given. Third, a brief historical sketch of ACs is presented. The fourth part of this chapter reviews recent developments in AC practice, including the increasing use of developmental ACs, cross-cultural applications, and innovations in assessment methods employing computer and web-based technology. The final part discusses AC research.
ASSESSMENT CENTERS: OVERVIEW OF PRACTICE AND RESEARCH
For five years Paul has been quite a successful salesman. However, when he reads a job posting of a district sales manager, he decides to apply for this position. He sends his resume and is invited for a background interview. Additionally and much to his surprise, he is required to participate in an AC. First, he must sell a product to a role-player, who plays the role of client. Next, he is confronted with the in-tray of a manager and has to handle various letters, memos, and announcements, and all of this against the clock. Finally, Paul participates together with other candidates for the job in a discussion. In this group discussion Paul and the other applicants have to reach consensus on which sales strategy an organization must adopt in the next year. While performing in these three exercises, Paul is observed and evaluated by one or two trained individuals. Afterwards Paul learns that he is not selected for the job. He also receives feedback about his performance in the AC. Undoubtedly, his salesmanship skills (e.g., sensitivity for a client’s needs, ability to handle objections) are excellent. However, his managerial qualities such as planning and organization are still underdeveloped.
This short example illustrates one of the added values of ACs vis-à-vis background interviews or biodata. Whereas interviews and biographical inventories tend to look back, ACs enable organizations to get a picture of the managerial potential of candidates in situations which differ from their actual job. In the last forty years, this AC method has become a popular approach for the selection and assessment of managers in Western Europe and the United States of America (Thornton & Krause, 2009). The following text provides an introduction to ACs. The basic features are exemplified and their history is briefly portrayed. We also aim to give a fine-grained picture of the advantages and disadvantages of ACs by confronting 7 popular slogans of ACs with recent empirical evidence.
Basic Features of Assessment Centers
Although the makeup of an AC varies and ACs exhibit considerable differences, there are several common features. The following elements are essential requirements of a program to be called an AC (International Task Force on Assessment Center Guidelines, 2000, p. 316).
First, a job analysis must be conducted to determine the dimensions for effective job performance. For instance, examples of important dimensions for a managerial job may include among others sensitivity, delegating, planning, decision making, oral communication, sociability, vision, problem solving, initiative, leadership, stress tolerance, etc.
Second, multiple exercises must be developed. These exercises sample essential situations of the target job and elicit information for rating the dimensions.
Third, although personality questionnaires, background interviews, and cognitive ability tests are often included in ACs, situational exercises and simulations serve as the most prominent feature. Below we present some examples of these AC exercises.
Fourth, several candidates (also known as assessees) meet on the same day at the same location to participate in these exercises.
Fifth, multiple raters (also known as assessors) should be used for each candidate. Several trained raters, observe the behavior of the candidates in the various exercises, record their observations and evaluate assessees on the dimensions. Both psychologists and line managers serve as assessors.
Sixth, assessors are trained to a performance standard. For instance, they learn to systematically record behavior and to classify their behavioral observations into meaningful and relevant categories (i.e., dimensions).
Finally, the integration of behaviors should rely on pooling of information. After viewing all candidates, assessors meet to integrate their observations and ratings and to reach consensus about the candidates. This integration of results can also be done by means of a statistical integration process.
Some Examples of Assessment Center Exercises
As mentioned above, a basic feature of ACs is that situational exercises are used. These situational exercises simulate some important aspects of the target job. Each situational exercise allows assessors to observe and rate the assessees on the various dimensions. Below six popular situational exercises are presented (Thornton & Mueller-Hanson, 2004; Thornton & Rupp, 2006).
First, two popular individual exercises are explained. The “in basket (or in-tray)” exercise simulates a manager’s in-tray. If possible, real in-tray items are used such as memos, letters, reports, etc. Participants are asked to stand in for the manager and handle the in-tray. In a short time-span and under pressure they have to decide on their actions. Usually, assessees explain the rationale for their actions in a subsequent interview.
Another popular individual exercise is the “written case analysis”. Here the assessee reads material that depicts an organizational problem. For instance, based on a large amount of financial information, assessees have to write down their conclusions.
Second, also several one-on-one exercises are often used in assessment centers. “Fact finding” is an exercise that confronts the participants with a problem. Based on a small amount of information about this problem (e.g., a faulty product, etc.) the participant is given the opportunity to acquire additional information by asking specific questions to a panel of resource persons. Afterwards the participant proposes his/her final recommendations to solve the problem.
During the “oral presentation” exercise, the candidates are instructed to deliver a speech about a particular topic. It is also possible that they give a more formal presentation about the case material of another exercise. After the presentation, each candidate is typically challenged by (an) assessor(s).
In the “role-play” the participant talks one-to-one with a role-player (acting out a colleague, a capable but disgruntled subordinate, a client, or an irate customer) and attempts to resolve the problem. This role-player should play his/her role consistently from candidate to candidate.
Finally, also group exercises are popular in assessment centers. In the “group discussion” exercise several candidates are asked to resolve a particular task. For instance, candidates have to reach consensus on which marketing strategy to follow. Group discussions may be competitive (i.e., each candidate is assigned a specific role) or cooperative (i.e., there are no roles assigned to candidates).
Design of Assessment Centers
Thornton and Byham (1982) specify that practitioners should proceed through the following 11 stages when constructing and implementing an AC:
• Identify the purpose of the AC (e.g., selection/promotion, development, training).
• Identify the participants (e.g., through self-nomination, through supervisor nomination, through pre-screening tests, etc.).
• Conduct a thorough job analysis of the target job.
• Derive a set of dimensions, thought to be necessary for effective performance on the target job.
• Develop exercises to afford measurement of these dimensions.
• Generate an exercise by dimension matrix (i.e., to indicate which dimensions are measured per exercise).
• Select and train assessors.
• Observe and record behavior, classify behaviors in dimensional terms and provide dimensional ratings.
• Assign an overall assessment rating (O.A.R.) to candidates (e.g., select, reject, promote, develop, etc.).
• Provide feedback to candidates.
• Report back to management on recommended actions.
Brief History of Assessment Centers
The first ACs were developed in a military context. After World War I the Germans decided to modify their officer selection system. In the new system over a three-day period officers were given psychological tests and various naturalistic exercises. A multidisciplinary board of assessors (i.e., psychologists, physicians, officers) rated the officers on a global ‘leadership’ dimension.
After observing the German programs, the ‘British War Office Selection Boards’ also began to utilize ACs for selection of World War II officers. Besides psychiatric interviews and a series of psychological tests, the officers were required to proceed through a number of realistic group and individual simulations.
Another important predecessor of ACs was Henry Murray’s ‘Harvard Psychological Clinic Study’. In this study Murray and his colleagues employed methods relevant to later AC developments, for instance the focus on observed behavior.
During the second World War ACs began to be used in the United States. In particular, the Office of Strategic Services (O.S.S.) attempted to improve their intelligence agent selection. The development of this ‘AC’ was modeled on Murray’s ‘Harvard Psychological Clinic Study’. The 3-day AC program of the O.S.S., which started in December 1943, consisted of interviews, psychological tests, leaderless groups, several of an outdoor nature (‘Brook exercise’, ‘Wall exercise’, ‘Construction exercise’, ‘Obstacle exercise‘), assigned leadership roles, stress interviews and many others. Over 7000 candidates were assessed.
In 1945 ‘The British Civil Service Commission’ used the AC technique for the first time in a non-military context. In the United States American Telegraph & Telephone Company (AT&T) was the first company to use ACs on a large scale. Psychologists served as assessors and evaluated each candidate on 25 dimensions.
In 1958, the Michigan Bell Personnel Assessment Program was conducted. This was the first industrial application. Instead of psychologists, this program was staffed by internal company managers. Other features were the use of a thorough job analysis to derive the dimensions, the standardized rating and consensus process, the removal of motivational and personality tests, and the extensive assessor training given.
In the sixties, the AC approach spread very slowly, as only a small number of large organizations were adopting it. Examples are IBM, Sears, Standard Oil (Ohio), General Electric and Penney. In 1969 twelve US organizations developed assessment programs. The real expansion of ACs began in the seventies. By the end of the decade, over 1000 organizations established AC programs.
By the end of the seventies the AC method spread internationally. Other developments included that besides selection ACs were being increasingly used for developmental purposes such as early identification of potential, management development or training. The target population expanded from M(anagerial) S(enior) E(xecutive) P(rofessional)-functions to students, engineers, salespeople, military, social workers, clerks and production workers.
Today, ACs are fairly common in large organizations in the United States, Australia, South Africa, and Europe. Its popularity in Asia is also growing.
Recent Developments in Assessment Center Practice
Applying Assessment Centers in Multi-national and Cross-cultural Settings
A first recent development in AC practice is the increased use of ACs in international applications. In fact, the spread of ACs around the world, the cross cultural applications of ACs, the globalization of businesses, the need for global executives, and the establishment of consultancies offering AC services in many other countries have raised questions about the application of assessment practices in diverse countries. Are ACs useful in selecting persons from a home country to serve in another country? Along these lines, Briscoe (1997) suggested that careful attention should be paid to the design of other exercises, the use of different dimensions, the use of assessors from both the home and host country, the evaluation of behaviors, and the provision of feedback. However, many of these recommendations still have to be put into practice.
In the only empirical, predictive validity study on this topic of which we are aware, Lievens, Harris, Van Keer, and Bisqueret (2003) developed and validated an AC for selecting European managers for a cross-cultural training program in Japan. Besides AC exercises, the procedure included cognitive ability and personality tests and a behavior description interview. The dimensions of adaptability, team work, and communication as measured by a group discussion exercise emerged as valid predictors, beyond cognitive ability and personality tests. Dimensions measured in a presentation did not emerge as significant predictors, showing that exercise design is an important issue in ACs for international applications.
We predict that ACs will be used more frequently in international settings. This will occur in three different ways. Home-country organizations will use ACs to assess persons going to host-countries. Home-country organizations will use their assessment methods to assess host-country persons in those other countries. Organizations in countries not currently using ACs will adopt the method. Each of these applications of the AC method presents unique challenges. AC proponents and adopters will have to make choices about what elements and specific practices of the method can be kept the same from their points of origin to the new location, and what adaptations need to be made to accommodate the unique aspects of the new location.
Use of Technology and Virtual Assessment Centers
The availability of computers and electronic media has provided the opportunity to increase the use of technology in ACs. Initially, computers were used to compile and analyze ratings from a team of assessors. Recently, more sophisticated applications have emerged, primarily in the methods to present stimuli. Exercise stimuli have been presented via video monitors and on computer-based simulations. Other applications of technology involve capturing behavior on audio and video recordings, sometimes from remote locations. These recordings can then be analyzed in a traditional manner by trained observers, or by using sophisticated software programs. Other assessment programs have used the web to capture electronic records of various achievements including text, audio, and video media. Automated analysis of written responses can evaluate the content and quality of writing samples. In addition, software can analyze voice tone. Special software has been developed to automate the process of writing reports. Furthermore, the web can be used to facilitate all stages of an assessment process including administration, exercise delivery, scoring, data tracking, report writing, and feedback.
A number of these technological developments increase the fidelity of exercise in terms of the stimuli presented to the participant (e.g., managers nowadays typically receive information via electronic media and respond online). Thus, in our estimation, high technology in an exercise may increase the realism of the exercise. Other aspects of high tech assessment exercises may in fact decrease the fidelity of the assessment, especially response fidelity. For example, some computerized in-baskets call for the participant to respond by choosing among a number of pre-established alternatives. In real-life, managers do not typically have the alternatives presented. In fact, they must generate alternatives and then overtly write a response. In some exercises, a video depicts a subordinate’s comments and the participant selects among a set of pre-established responses. This sort of assessment method does not have fidelity with dynamic interpersonal interactions. Computerized in-baskets and video-based assessment techniques may have predictive validity, but they are qualitatively different from the overt behaviors required in the typical interpersonal and decision making simulations that have been the hallmark of the AC method.
Assessment Centers for Developmental Purposes
The most pronounced trend in AC practice in recent years is the shift in their predominant purpose from selection/promotion to development. This increased interest in using ACs to develop the talent of managers remaining in their current positions results among other things from the flattening and downsizing of organizations and the fewer promotional opportunities available.
There are several variants of developmental ACs. In some, the emphasis is on the diagnosis of training needs of individuals. The design of these centers, including the dimensions and exercises, is very similar to promotional centers. Another variant is a training center in which the objective is to foster skill development. To turn the program into a learning experience, steps are taken to provide immediate feedback, practice, reinforcement of learning, transfer of training, and follow-up developmental support in the organization. A third variant of developmental ACs are programs designed to promote development of organizational units. The use of simulation technology for development purposes typically involves the assessment of intact work groups participating in complex organization games.
Developmental ACs have become quite popular, but have met with numerous challenges. One of the primary challenges is to provide evidence that the program has some impact on participants. Impact may take the form of a) intentions to take action to develop, b) engagement in some form of developmental experience, c) change of understanding of the performance dimensions, d) improvement in skills, e) change of behavior on the job, or f) improvement in organizational effectiveness. Jones and Whitmore (1995) found that career advancement of assessed and non-assessed managers did not differ, except when the assessed managers engaged in developmental activities. Unfortunately, most managers do not follow up AC diagnoses with developmental activities. Positive effects do not automatically ensue and are likely to occur only if there are a number of other support systems in place in the organization to help the assessee after the AC experience. Woo, Simms, Rupp, and Gibbons (2008) studied the factors that may influence participants’ behavioral engagement in developmental activities (programs) following feedback. Their results showed that higher assessor ratings were associated with higher engagement. They also reported that to the extent that participants rated themselves lower than did assessors, program engagement was higher.
Research Evidence on Assessment Centers
Criterion-Related Validity
Many AC studies investigated the validity of ACs. Recently, Arthur, Day, McNelly, and Edens (2003) conducted a meta-analysis (large-scale evaluation) of these studies. They focused on the criterion-related validity (this is a measure of how well ACs predict an outcome, for instance job performance). A typical way to achieve this is in relation to the extent to which a score on a personality of AC dimensions. Criterion-related validities varied from .25 to .39. Other studies have provided evidence that the criterion-related validity of ACs holds across jobs, time, and contexts. For example, Jansen and Stoop (2001) demonstrated the validity of ACs in the long run. They validated an AC over a 7-year period with average salary growth as the criterion. The validity of the overall assessment rating was .39. Recent research also showed that ACs explain additional validity over and above cognitive ability tests and personality. For instance, the recent meta-analysis of Meriac, Hoffman, Woehr, and Fleisher (2008) suggested that assessment center dimensions explain a substantial proportion of variance in job performance beyond cognitive ability and personality.
Reliability
Over the years, various studies have also examined the inter-rater reliability of assessors in ACs. These studies investigate whether assessors are consistent in their ratings vis-à-vis each other. Generally, this body of research shows a relatively positive picture (Lievens, 2002). Inter-rater reliability is highest for the overall assessment ratings of assessors. When different assessors are asked to evaluate a candidate on a given dimension within an exercise, they tend to agree only moderately. Undoubtedly, amount of training is an important contributing factor here because inter-rater reliabilities for trained assessors are significantly higher from these of untrained assessors. Nonetheless, it is important that assessor training goes beyond behavior observation training. This training focuses on strategies to improve observation. More emphasis should be placed on familiarizing assessors with the rating dimensions and providing them with the same evaluative standards as a reference for judging performance, as is the case in frame-of-reference training (Lievens, 2001).
Utility
One of the widespread criticisms of ACs is related to their high costs. Utility studies can put this criticism to the test as they examine the degree to which a selection procedure improves the quality of applicants chosen over random selection. Research on the utility of AC (e.g., Hoffman & Thornton, 1997) yields quite a consistent picture. ACs seem to be superior visa versa more traditional selection techniques such as the interview and paper-and-pencil tests. In other words, the benefits of ACs in terms of selecting high-performing managers outweigh the design and administration costs of ACs.
Adverse Impact
A widely held view is that there exists no bias or potential for unfair discrimination in ACs. Do particular subgroups (e.g., whites, men, younger people) systematically receive higher ratings in ACs? Recently, Dean, Bobko, and Roth (2008) suggested in their meta-analysis that assessment centers may be associated with more adverse impact against Blacks than was previously mentioned in the literature. However, they also concluded that assessment centers may have less adverse impact for Hispanics and females. In terms of age, Clapham and Fulford (1997) found that older candidates (older than 40 years) received significantly lower ratings than younger candidates.
Applicant Reactions
Generally, research attests to the high acceptability of ACs. Participants react positively to their AC experiences, although low-scorers should be given more attention. Apparently, applicants react positively to ACs, because centers include work samples and simulations (Hausknecht, Day, & Thomas, 2004). Research identified the perceived job relatedness to be the main determinant of the favorable reactions to these simulation-based techniques.
Coachability and Fakability
As noted by Lievens (2002), the relatively small body of research on the trainability of AC performance shows that assessees may benefit from training and experience when taking individual exercises such as in-baskets. The effects of training and experience on subsequent performance in more unstructured group exercises are less conclusive. Research on “faking good” in ACs is even scarcer. In the only study that we are aware of, McFarland, Ryan, and Kriska (2003) compared impression management in ACs to impression management in structured interviews. They found less use of candidate impression management tactics in an AC exercise (a role-play) than in a situational interview. Apparently, candidates are already so busy acting out their designated role-play character that they have little cognitive resources left to engage in impression management.
Construct-related Validity
One of the consistent findings in the AC field has been that there are low correlations among ratings of a single dimension across exercises (i.e., weak convergent validity) and high correlations among ratings of various dimensions within one exercise (i.e., weak discriminant validity). Recent research shows that these results are not really troublesome for ACs (Lance, 2008; Lievens, 2009). An AC is composed of several exercises which are carefully selected to cover specific job-related competences. Consequently, they place different psychological demands on the assessees. Therefore, it should be no surprise that performance in ACs is highly exercise-specific. For instance, one might expect an assessee to behave differently – even inconsistently - in a one-to-one role-play as compared to in a group discussion.
Conclusions
The AC method continues to be used in a variety of organizational settings and to generate numerous research studies. In this chapter, the AC approach was presented and exemplified. Recent developments in AC practice in the past few years included the increasing use of developmental ACs, cross-cultural applications, and innovations in assessment methods employing computer and web-based technology. We also reviewed empirical research on ACs. This state-of-the-art exemplified that ACs score well on ‘hard’ validity criteria such as consistent evidence of criterion-related validity, good utility and inter-rater reliability. Additionally, reasons for AC popularity include also more ‘soft’ benefits such as inherent fairness and enthusiast candidate reactions.
References
Thornton, G. C. III, & Byham, W. C. (1982). Assessment centers and managerial performance. New York: Academic Press.
Thornton, G. C.III, & Mueller-Hanson, R. A. (2004). Developing organizational simulations: A guide for practitioners and students. Mahwah, NJ: Erlbaum.
Thornton, G. C., III, & Rupp, D. (2006). Assessment centers in human resource management: Strategies for prediction, diagnosis, and development. Mahwah, NJ: Erlbaum.
Arthur, W., Jr., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125-154.
Briscoe, D.R. (1997). Assessment centers: Cross-cultural and cross-national issues. Journal of Social Behavior and Personality, 12, 261-270.
Clapham, M. M, & Fulford, M. D (1997). Age bias in assessment center ratings. Journal of Managerial Issues, 9, 373-87.
Dean, M. A., Bobko, P., & Roth, P. L. (2008). Ethnic and gender subgroup differences in assessment center ratings: A meta-analysis. Journal of Applied Psychology, 93, 685-691.
Hausknecht, J. P., Day, D. V., & Thomas, S. C. (2004). Applicant reactions to selection procedures: An updated model and meta-analysis. Personnel Psychology, 57, 639-683.
Hoffman, C. C., & Thornton, G. C. (1997). Examining selection utility where competing predictors differ in adverse impact. Personnel Psychology, 50, 455-470.
International Task Force on Assessment Center Guidelines (2000). Guidelines Ethical Considerations for Assessment Center Operations. Public Personnel Management, 29, 315-331.
Jansen, P. G. W., & Stoop, B. A. M. (2001). The dynamics of assessment center validity: Results of a 7-year study. Journal of Applied Psychology, 86, 741-753.
Jones, R. G., & Whitmore, M. D. (1995). Evaluating developmental assessment centers as interventions. Personnel Psychology, 48, 377-388.
Lance, C. E. (2008). Why assessment centers (ACs) don't work the way they're
supposed to. Industrial and Organizational Psychology: Perspectives on
Science and Practice, 1, 87-100.
Lievens, F. (2001). Assessor training strategies and their effects on accuracy, inter-rater reliability, and discriminant validity. Journal of Applied Psychology, 86, 255-264.
Lievens, F. (2002). An examination of the accuracy of slogans related to assessment centres. Personnel Review, 31, 86-102.
Lievens, F. (2009). Assessment centers: A tale about dimensions, exercises, and dancing bears. European Journal of Work and Organizational Psychology, 18, 102-121.
Lievens, F., & Harris, M. M., Van Keer, E., & Bisqueret, C. (2003). Predicting cross-cultural training performance: The validity of personality, cognitive ability, and dimensions measured by an assessment center and a behavior description interview. Journal of Applied Psychology, 88, 476-489.
McFarland, L. A., Ryan, A. M., & Kriska, S. D. (2003). Impression management use and effectiveness across assessment methods. Journal of Management, 29, 641-661.
Meriac, J. P., Hoffman, B. J., Woehr, D. J., & Fleisher, M. S. (2008). Further evidence for the validity of assessment center dimensions: A meta-analysis of the incremental criterion-related validity of dimension ratings. Journal of Applied Psychology, 93, 1042-1052.
Thornton, G. C., & Krause, D. E. (2009). Selection versus development assessment centers: an international survey of design, execution, and evaluation. International Journal of Human Resource Management, 20, 478-498.
Woo, S. E., Sims, C. S., Rupp, D. E., & Gibbons, A. M. (2008). Development engagement within and following developmental assessment centers: Considering feedback favorability and self–assessor agreement. Personnel Psychology, 61, 727–759.
Questions for classroom discussion
1. What are the basic requirements of a program to be called an assessment center?
2. Give some popular assessment center exercises.
3. Why do we predict that assessment centers will be used more frequently in international contexts?
4. Give three different variants of developmental assessment centers.
5. What does previous research suggest about the inter-rater reliability of assessors in assessment centers?
6. Assessment centers are rather expensive. What is the evidence about the utility of assessment centers?
Author information
Filip Lievens is Professor at the Department of Personnel Management and Work and Organizational Psychology at Ghent University, Belgium. In 1999, he earned his Ph.D. from the same university. His current research interests focus on alternative selection procedures (e.g., assessment centers, situational judgment tests, web-based assessment) and organizational attractiveness). He is the author of over 70 articles, among others in the Journal of Applied Psychology, Personnel Psychology, Journal of Organizational Behavior, Journal of Occupational and Organizational Psychology, Applied Psychology: An international Review, and International Journal of Selection and Assessment. He has received several awards including the Distinguished Early Career Award from the Society of Industrial and Organizational Psychology (2006).
Correspondence concerning this chapter should be addressed to Filip Lievens,
Email: filip.lievens@ugent.be.
Web page: http://allserv.rug.ac.be/~flievens/
Eveline Schollaert is teaching assistant at the Department of Personnel Management and Work and Organizational Psychology at Ghent University, Belgium. Her research interests include recruitment and personnel selection (assessment centers, situational judgment tests, personality scales, etc.). She aims to obtain her Ph.D. in 2012.
Email: Eveline.Schollaert@UGent.be
