TESTING AND ASSESSMENT:
AN EMPLOYER’S GUIDE
TO GOOD PRACTICES

U.S. Department of Labor Employment and Training Administration

Brought to you by

Foreword

PURPOSE of the GUIDE

In today’s competitive marketplace and complex legal environment, employers face the challenge of attracting, developing, and retaining the best employees. Michael Eisner, CEO of the Disney Corporation, recognized the impact of personnel decisions on a business’ bottom-line when he remarked, “My inventory goes home every night.”

This guide is to help managers and human resource (HR) professionals use assessment practices that are the right choices for reaching their organizations’ HR goals. It conveys the essential concepts of employment testing in easy-to-understand terms so that managers and HR professionals can

Evaluate and select assessment tools/procedures that maximize chances for getting the right fit between jobs and employees
Administer and score assessment tools that are the most efficient and effective for their particular needs
Interpret assessment results in an accurate manner
Understand the professional and legal standards to be followed when conducting personnel assessment.

FORMAT of the GUIDE

This Guide is structured around a set of assessment principles and their applications. The information is organized so that readers from a variety of backgrounds will find the information presented in a clear and useful manner.

� Each chapter covers a critical aspect of the assessment process. The issues involved in each aspect are outlined at the beginning of each chapter.

� Thirteen principles of assessment are explained in the Guide. The last chapter (Chapter 9) summarizes the main points of the principles, serving as a review of the material discussed in the Guide.

� Appendix A offers a list of resource materials for those interested in more information on a particular topic, and Appendix B is a glossary for quick clarification of terms and concepts.

The Guide is designed to provide accurate and important information regarding testing as part of a personnel assessment program. It gives general guidelines and must not be viewed as legal advice.

Acknowledgments

Testing and Assessment: An Employer’s Guide to Good Practices (Guide) was produced and funded by the Skills Assessment and Analysis Program in the U.S. Department of Labor, Employment and Training Administration, Office of Policy and Research (OPR) under the direction of Gerard F. Fiala, Administrator. The Skills Assessment and Analysis Program is directed by Donna Dye, Personnel Research Psychologist, who provided technical direction and support for this Guide.

The Guide was prepared under Department of Labor Grants with the North Carolina Employment Security Commission, Southern Assessment Research and Development Center and National O*NET Consortium; the New York Department of Labor; and the Utah Department of Employment Security. The Guide was completed under the direction of David Rivkin. Mr. Rivkin also served as editor of the Guide. Authors of this Guide were Syed Saad, Gary W. Carter, Mark Rothenberg, and Enid Israelson. Grateful acknowledgment is made to Phil Lewis, Patrice Gilliam-Johnson, Jonathan Levine, and Brenda Dunn for their contribution. Thanks are also given to Ann Kump, Helen Tannenbaum, Don Kreger, Kristin Fiske, and Marilyn Silver whose valuable suggestions were very much appreciated. Grateful acknowledgment is also made to Suzan Chastain, Department of Labor, Office of the Solicitor, Division of Civil Rights, and Hilary R. Weinerand and Cynthia Misicka of the Equal Employment Opportunity Commission for consultant review and insights into the final preparation of this Guide.

Special Notice

Material in this publication is in the public domain and may be reproduced, fully or partially, without
permission of the federal government. Source credit is requested but not required. Permission is
required only to reproduce any copyrighted material contained herein.

This material will be made available to sensory impaired individuals upon request.
Voice phone: (202) 219-7664
TDD* phone: (800) 326-2577

*Telecommunications Device for the Deaf

Table of Contents

Chapter Page

1 Personnel Assessment ...................................... 1-1

2 Understanding the Legal Context of Assessment—Employment
Laws and Regulations with Implications for Assessment ............. 2-1

3 Understanding Test Quality—Concepts of Reliability and Validity ...... 3-1

4 Assessment Tools and Their Uses ............................. 4-1

5 How to Select Tests—Standards for Evaluating Tests ............... 5-1

6 Administering Assessment Instruments .......................... 6-1

7 Using, Scoring, and Interpreting Assessment Instruments ............ 7-1

8 Issues and Concerns with Assessment .......................... 8-1

9 A Review—Principles of Assessment ........................... 9-1

Appendix

A Sources of Additional Information on Personnel Assessment .......... A-1

B Glossary of Assessment Terms ................................ B-1

CHAPTER 1 Personnel Assessment

Personnel assessment is a systematic approach to gathering information about individuals. This information is used to make employment or career-related decisions about applicants and employees.

Assessment is conducted for some specific purpose. For example, you, as an employer, may conduct personnel assessment to select employees for a job. Career counselors may conduct personnel assessment to provide career guidance to clients.

Chapter Highlights

1. Personnel assessment tools: tests and procedures

2. Relationship between the personnel assessment process and tests and procedures

3. What do tests measure?

4. Why do organizations conduct assessment?

5. Some situations in which an organization may benefit from testing

6. Importance of using tests in a purposeful manner

7. Limitations of personnel tests and procedures—fallibility of test scores.

Principles of Assessment Discussed

Use assessment tools in a purposeful manner Use the whole-person approach to assessment.

1. Personnel assessment tools: tests and procedures

Any test or procedure used to measure an individual’s employment or career-related qualifications and interests can be considered a personnel assessment tool. There are many types of personnel assessment tools. These include traditional knowledge and ability tests, inventories, subjective procedures, and projective instruments. In this guide, the term test will be used as a generic term to refer to any instrument or procedure that samples behavior or performance.

Personnel assessment tools differ in

Purpose, e.g., selection, placement, promotion, career counseling, or training
What they are designed to measure, e.g., abilities, skills, work styles, work values, or vocational interests
What they are designed to predict, e.g., job performance, managerial potential, career success, job satisfaction, or tenure
Format, e.g., paper-and-pencil, work-sample, or computer simulation
Level of standardization, objectivity, and quantifiability— Assessment tools and procedures vary greatly on these factors. For example, there are subjective evaluations of resumes, highly structured achievement tests, interviews having varying degrees of structure, and personality inventories with no specific right or wrong answers.

All assessment tools used to make employment decisions, regardless of their format, level of standardization, or objectivity, are subject to professional and legal standards. For example, both the evaluation of a resume and the use of a highly standardized achievement test must comply with applicable laws. Assessment tools used solely for career exploration or counseling are usually not held to the same legal standards.

3. What do tests measure?

People differ on many psychological and physical characteristics. These characteristics are called constructs. For example, people skillful in verbal and mathematical reasoning are considered high on mental ability. Those who have little physical stamina and strength are labeled low on endurance and physical strength. The terms mental ability, endurance and physical strength are constructs. Constructs are used to identify personal characteristics and to sort people in terms of how much they possess of such characteristics.

Constructs cannot be seen or heard, but we can observe their effects on other variables. For example, we don’t observe physical strength but we can observe people with great strength lifting heavy objects and people with limited strength attempting, but failing, to lift these

Tests, inventories, and procedures are assessment tools that may be used to measure an individual’s abilities, values, and personality traits. They are

components of the assessment process.
!	observations	!	physical ability tests
!	resume evaluations	!	personality inventories
!	application blanks/questionnaires	!	honesty/integrity inventories
!	biodata inventories	!	interest inventories
!	interviews	!	work values inventories
!	work samples/performance tests	!	assessment centers
!	achievement tests	!	drug tests
!	general ability tests	!	medical tests
!	specific ability tests

Assessment process

Systematic approach to combining and evaluating all the information gained from testing and using it to make career or employment-related decisions.

Figure 1. Relationship between assessment tools and the assessment process.

objects. Such differences in characteristics among people have important implications in the employment context.

Employees and applicants vary widely in their knowledge, skills, abilities, interests, work styles, and other characteristics. These differences systematically affect the way people perform or behave on the job.

These differences in characteristics are not necessarily apparent by simply observing the employee or job applicant. Employment tests can be used to gather accurate information about job-relevant characteristics. This information helps assess the fit or match between people and jobs. To give an example, an applicant’s score on a mechanical test reflects his or her mechanical ability as measured by the test. This score can be used to predict how well that applicant is likely to perform in a job that requires mechanical ability, as demonstrated through a professionally conducted job analysis. Tests can be used in this way to identify potentially good workers.

Some tests can be used to predict employee and applicant job performance. In testing terms, whatever the test is designed to predict is called the criterion. A criterion can be any measure of work behavior or any outcome that can be used as the standard for successful job performance. Some commonly used criteria are productivity, supervisory ratings of job performance, success in training, tenure, and absenteeism. For example, in measuring job performance, supervisory ratings could be the criterion predicted by a test of mechanical ability. How well a test predicts a criterion is one indication of the usefulness of the test.

4. Why do organizations conduct assessment?

Organizations use assessment tools and procedures to help them perform the following human resource functions:

� Selection. Organizations want to be able to identify and hire the best people for the job and the organization in a fair and efficient manner. A properly developed assessment tool may provide a way to select successful sales people, concerned customer service representatives, and effective workers in many other occupations.

� Placement. Organizations also want to be able to assign people to the appropriate job level. For example, an organization may have several managerial positions, each having a different level of responsibility. Assessment may provide information that helps organizations achieve the best fit between employees and jobs.

� Training and development. Tests are used to find out whether employees have mastered training materials. They can help identify those applicants and employees who might benefit from either remedial or advanced training. Information gained from testing can be used to design or modify training programs. Test results also help individuals identify areas in which self-development activities would be useful.

� Promotion. Organizations may use tests to identify employees who possess managerial potential or higher level capabilities, so that these employees can be promoted to assume greater duties and responsibilities.

� Career exploration and guidance. Tests are sometimes used to help people make educational and vocational choices. Tests may provide information that helps individuals choose occupations in which they are likely to be successful and satisfied.

� Program evaluation. Tests may provide information that the organization can use to determine whether employees are benefiting from training and development programs.

7. Limitations of personnel tests and procedures—fallibility of test scores

Professionally developed tests and procedures that are used as part of a planned assessment program may help you select and hire more qualified and productive employees. However, it is essential to understand that all assessment tools are subject to errors, both in measuring a characteristic, such as verbal ability, and in predicting performance criteria, such as success on the job. This is true for all tests and procedures, regardless of how objective or standardized they might be.

� Do not expect any test or procedure to measure a personal trait or ability with perfect accuracy for

every single person. � Do not expect any test or procedure to be completely accurate in predicting performance.

There will be cases where a test score or procedure will predict someone to be a good worker, who, in fact, is not. There will also be cases where an individual receiving a low score will be rejected, who, in fact, would actually be capable and a good worker. Such errors in the assessment context are called selection errors. Selection errors cannot be completely avoided in any assessment program.

Why do organizations conduct testing despite these errors? The answer is that appropriate use of professionally developed assessment tools on average enables organizations to make more effective employment-related decisions than use of simple observations or random decision making.

Using a single test or procedure will provide you with a limited view of a person’s employment or career-related qualifications. Moreover, you may reach a mistaken conclusion by giving too much weight to a single test result. On the other hand, using a variety of assessment tools enables you to get a more complete picture of the individual. The practice of using a variety of tests and procedures to more fully assess people is referred to as the whole-person approach to personnel assessment. This will help reduce the number of selection errors made and will boost the effectiveness of your decision making. This leads to an important principle of assessment.

Principle of Assessment

Do not rely too much on any one test to make decisions. Use the whole-person approach to assessment.

CHAPTER 2 Understanding the Legal Context of Assessment—Employment Laws and Regulations with Implications for Assessment

The number of laws and regulations governing the employment process has increased over the past four decades. Many of these laws and regulations have important implications for conducting employment assessment. This chapter discusses what you should do to make your practices consistent with legal, professional, and ethical standards.

Chapter Highlights

1. 1. Title VII of the Civil Rights Act (CRA) of 1964, as amended in 1972; Tower Amendment to Title VII

2. 2. Age Discrimination in Employment Act of 1967 (ADEA)

3. 3. Equal Employment Opportunity Commission (EEOC) - 1972

4. 4. Uniform Guidelines on Employee Selection Procedures - 1978; adverse or disparate impact, approaches to determine existence of adverse impact, four-fifths rule, job-relatedness, business necessity, biased assessment procedures

5. 5. Title I of the Civil Rights Act (CRA) of 1991

6. 6. Americans with Disabilities Act (ADA) - 1990

7. 7. Record keeping of adverse impact and job-relatedness of tests

8. 8. The Standards for Educational and Psychological Testing¹ - 1985; The Principles for the Validation and Use of Personnel Selection Procedures - 1987

9. 9. Relationship between federal, state, and local employment laws.

Principles of Assessment Discussed

Use only assessment instruments that are unbiased and fair to all groups.

The general purpose of employment laws and regulations is to prohibit unfair discrimination in employment and provide equal employment opportunity for all. Unfair discrimination occurs when employment decisions are based on race, sex, religion, ethnicity, age, or disability rather than on job

¹Currently under revision by the American Psychological Association.

relevant knowledge, skills, abilities, and other characteristics. Employment practices that unfairly discriminate against people are called unlawful or discriminatory employment practices.

The summaries of the laws and regulations in this chapter focus on their impact on employment testing and assessment. Before you institute any policies based on these laws and regulations, read the specific laws carefully, and consult with your legal advisors regarding the implications for your particular assessment program.

1. Title VII of the Civil Rights Act (CRA) of 1964 (as amended in 1972); Tower Amendment to Title VII

Title VII is landmark legislation that prohibits unfair discrimination in all terms and conditions of employment based on race, color, religion, sex, or national origin. Other subsequent legislation, for example, ADEA and ADA, has added age and disability, respectively, to this list. Women and men, people age 40 and older, people with disabilities, and people belonging to a racial, religious, or ethnic group are protected under Title VII and other employment laws. Individuals in these categories are referred to as members of a protected group. The employment practices covered by this law include the following:

. • recruitment • hiring • job classification

. • transfer • training • promotion

. • performance appraisal • compensation • union or other membership

. • disciplinary action • termination • fringe benefits.

Employers having 15 or more employees, employment agencies, and labor unions are subject to this law.

The Tower Amendment to this act stipulates that professionally developed workplace tests can be used to make employment decisions. However, only instruments that do not discriminate against any protected group can be used. Use only tests developed by experts who have demonstrated qualifications in this area.

3. Equal Employment Opportunity Commission (EEOC)—1972

The EEOC is responsible for enforcing federal laws prohibiting employment discrimination, including Title VII, the U. S. Environmental Protection Agency (EPA), the ADEA, and the ADA. It receives, investigates, and processes charges of unlawful employment practices of employers filed by an individual, a group of individuals, or one of its commissioners. If the EEOC determines that there is “reasonable cause” that an unlawful employment practice has occurred, it is also authorized to sue on behalf of the charging individual(s) or itself. The EEOC participated in developing the Uniform Guidelines on Employee Selection Procedures.

4. Uniform Guidelines on Employee Selection Procedures—1978; adverse or disparate impact, approaches to determine existence of adverse impact, four-fifths rule, job-relatedness, business necessity, biased assessment procedures

In 1978, the EEOC and three other federal agencies—the Civil Service Commission (predecessor of the Office of Personnel Management) and the Labor and Justice Departments—jointly issued the Uniform Guidelines on Employee Selection Procedures. The Guidelines incorporate a set of principles governing the use of employee selection procedures according to applicable laws. They provide a framework for employers and other organizations for determining the proper use of tests and other selection procedures. The Guidelines are legally binding under a number of civil rights laws, including Executive Order 11246 and the Civil Rights Requirements of the National Job Training Partnership Act and the Wagner Peyser Act. In reviewing the testing practices of organizations under Title VII, the courts generally give great importance to the Guidelines’ technical standards for establishing the job-relatedness of tests. Also, federal and state agencies, including the EEOC, apply the Uniform Guidelines in enforcing Title VII and related laws.

The Guidelines cover all employers employing 15 or more employees, labor organizations, and employment agencies. They also cover contractors and subcontractors to the federal government and organizations receiving federal assistance. They apply to all tests, inventories and procedures used to make employment decisions. Employment decisions include hiring, promotion, referral, disciplinary

action, termination, licensing, and certification. Training may be included as an employment decision if it
leads to any of the actions listed above. The Guidelines have significant implications for personnel
assessment.

One of the basic principles of the Uniform Guidelines is that it is unlawful to use a test or selection
procedure that creates adverse impact, unless justified. Adverse impact occurs when there is a
substantially different rate of selection in hiring, promotion, or other employment decisions that work to
the disadvantage of members of a race, sex, or ethnic group.

Different approaches exist that can be used to determine whether adverse impact has occurred.
Statistical Techniques may provide information regarding whether or not the use of a test results in
adverse impact. Adverse impact is normally indicated when the selection rate for one group is less than
80% (4/5) that of another. This measure is commonly referred to as the four-fifths or 80% rule.
However, variations in sample size may affect the interpretation of the calculation. For example, the
80% rule may not be accurate in detecting substantially different rates of selection in very large or small
samples. When determining whether there is adverse impact in very large or small samples, more
sensitive tests of statistical significance should be employed.

When there is no charge of adverse impact, the Guidelines do not require that you show the job-
relatedness of your assessment procedures. However, you are strongly encouraged to use only job-
related assessment tools.

If your assessment process results in adverse impact, you are required to eliminate it or justify its
continued use. The Guidelines recommend the following actions when adverse impact occurs:

� Modify the assessment instrument or procedure causing adverse impact.
� Exclude the component procedure causing adverse impact from your assessment program.
� Use an alternative procedure that causes little or no adverse impact, assuming that the alternative

procedure is substantially equally valid. � Use the selection instrument that has adverse impact if the procedure is job related and valid for

selecting better workers, and there is no equally effective procedure available that has less adverse

impact.

Note that for the continued use of assessment instruments or procedures that cause adverse impact, courts have required justification by business necessity as well as validity for the specific use. The issue of business necessity is specifically addressed in Title I of the Civil Rights Act of 1991 (see next section).

An assessment procedure that causes adverse impact may continue to be used only if there is evidence that

� It is job-related for the position in question. � Its continued use is justified by business necessity.

Demonstrating job-relatedness of a test is the same as establishing that the test may be validly used as desired. Chapter 3 discusses the concept of test validity and methods for establishing the validity or job-relatedness of a test.

Demonstrating the business necessity of using a particular assessment instrument involves showing that its use is essential to the safe and efficient operation of the business and there are no alternative procedures available that are substantially equally valid to achieve the business objectives with a lesser adverse impact.

Another issue of importance discussed in the Uniform Guidelines relates to test fairness. The Uniform Guidelines define biased or unfair assessment procedures as those assessment procedures on which one race, sex, or ethnic group characteristically obtains lower scores than members of another group and the differences in the scores are not reflected in differences in the job performance of members of the groups.

The meaning of scores on an unfair or biased assessment procedure will differ depending on the group membership of the person taking the test. Therefore, using biased tests can prevent employers from making equitable employment decisions. This leads to the next principle.

Principle of Assessment

Use only assessment instruments that are unbiased and fair to all groups.

Use of biased tools may result in unfair discrimination against members of the lower scoring groups. However, use of fair and unbiased tests can still result in adverse impact in some cases. If you are developing your own test or procedure, expert help may be advisable to make sure your procedure is fair to all relevant groups. If you are planning to purchase professionally developed assessment tools, first evaluate the fairness of those you are considering by reading the test manuals and consulting independent reviews.

6. Americans with Disabilities Act (ADA) - 1990

Under the ADA, qualified individuals with disabilities must be given equal opportunity in all aspects of employment. The law prohibits employers with 15 or more employees, labor unions, and employment agencies from discriminating against qualified individuals with disabilities. Prohibited discrimination includes failure to provide reasonable accommodation to persons with disabilities when doing so would not pose undue hardship.

A qualified individual with a disability is one who can perform the essential functions of a job, with or without reasonable accommodation.

� Disability is defined broadly to include any physical or mental impairment that substantially limits one or more of an individual’s major life activities, such as caring for oneself, walking, talking, hearing, or seeing. Some common examples include visual, speech, and hearing disabilities; epilepsy; specific learning disabilities; cancer; serious mental illness; AIDS and HIV infection; alcoholism; and past drug addiction. Noteworthy among conditions not covered are current illegal use of drugs, sexual behavior disorders, compulsive gambling, kleptomania, and pyromania.

� Essential functions are the primary job duties that are fundamental, and not marginal to the job. Factors relevant to determining whether a function is essential include written job descriptions, the amount of time spent performing the function, the consequences of not requiring the function, and the work experiences of employees who hold the same or similar jobs.

� Reasonable accommodation is defined as a change in the job application and selection process, a change in the work environment or the manner in which the work is performed, that enables a qualified person with a disability to enjoy equal employment opportunities. Under this Act, qualified individuals with disabilities must be provided reasonable accommodation so they can perform the essential job functions, as long as this does not create undue hardship to the employer.

� Undue hardship is defined as significant difficulty or additional expense and is determined based on a number of factors. Some factors that are considered are the nature and net cost of the accommodation, the financial resources of the facility, the number employed at the facility, the effect on resources and operations, the overall financial resources of the entire organization, and the fiscal relationship of the facility with the organization. An accommodation that is possible for a large organization may pose an undue hardship for a small organization.

The ADA has major implications for your assessment practices.

� In general, it is the responsibility of the individual with a disability to inform you that an accommodation is needed. However, you may ask for advance notice of accommodations required, for the hiring process only, so that you may adjust your testing program or facilities appropriately. When the need for accommodation is not obvious, you may request reasonable documentation of the applicant’s disability and functional limitations for which he or she needs an accommodation.

� Reasonable accommodation may involve making the test site accessible, or using an alternative assessment procedure. Administering employment tests to individuals with disabilities that require those individuals to use their impaired abilities is prohibited unless the tests are intended to measure one of these abilities. For example, under the ADA, when a test screens out one or more individuals with a disability, its use must be shown to be job-related for the position in question and justified by business necessity.

� One possible alternative procedure, if available, would be to use a form of the test that does not require use of the impaired ability. Another possibility is to use a procedure that compensates for the impaired ability, if appropriate. For example, allowing extra time to complete certain types of employment tests for someone with dyslexia or other learning disability, or providing a test with larger print or supplying a reader to a visually impaired individual where appropriate, would be considered reasonable accommodation.

� The ADA expressly prohibits making medical inquiries or administering medical examinations prior to making a job offer. Before making medical inquiries, or requiring medical exams, you must make an offer of employment to the applicant. You may make medical inquiries or require medical exams of an employee only when doing so is work-related and justified by business necessity. All medical information you obtain about your applicants and employees is strictly confidential and must be treated as such. Access to and use of this information is also greatly restricted. For a more detailed discussion of medical examinations see Chapter 4.

Your organization should develop a written policy on conducting testing and assessment of individuals with disabilities. This will help ensure compliance with the provisions of the ADA.

If you need assistance in complying with the ADA, there are several resources you may contact.

� The Job Accommodation Network: (800) 526-7234 � Industry-Labor Council on Employment and Disability: (516) 747-6323 � The American Foundation for the Blind: (202) 408-0200, (800) 232-5463 � The President’s Committee on Employment of People with Disabilities: (202) 376-6200 � Disability and Business Technical Assistance Centers: (800) 949-4232.

7. Record keeping of adverse impact and job-relatedness of tests

The Uniform Guidelines and subsequent regulations² require that all employers maintain a record of their employment-related activities, including statistics related to testing and adverse impact. Filing and record-keeping requirements for large employers (those with over 100 employees) are generally more extensive than those for employers with 100 or fewer employees. To learn more about the specific requirements, refer to EEOC regulations on record-keeping and reporting requirements under Title VII, and the ADA, 29 CFR part 1602, and the Uniform Guidelines.

8. The Standards for Educational and Psychological Testing - 1985; The Principles for the Validation and Use of Personnel Selection Procedures—1987

There are two resource guides published by major organizations in the testing field that will help you set up and maintain an assessment program. The principles and practices presented in these publications set the standards for professional conduct in all aspects of assessment.

� The Standards for Educational and Psychological Testing. This publication was developed jointly by the American Psychological Association (APA), the National Council on Measurement in Education (NCME), and the American Educational Research Association (AERA). The Standards are an authoritative and comprehensive source of information on how to develop, evaluate, and use tests and other assessment procedures in educational, employment, counseling, and clinical settings. Although developed as professional guidelines, they are consistent with applicable regulations and are frequently cited in litigation involving testing practices.

²29 CFR part 1602, as amended by 56 Fed. Reg. 35,753 (July 26, 1991); previously, record-keeping requirements did not apply to temporary and seasonal positions.

� The Principles for the Validation and Use of Personnel Selection Procedures. This publication was developed by the Society for Industrial and Organizational Psychology (SIOP). Like the Standards, the Principles are also an excellent guide to good practices in the choice, development, evaluation, and use of assessment tools. However, their main focus is on tools used in the personnel assessment context. The Principles explain their relationship to the Standards in the following way:

The Standards primarily address psychometric issues while the Principles primarily address the problems of making decisions in employee selection, placement, promotion, etc. The major concern of the Standards is general; the primary concern of the Principles is that performance on a test . . . is related to performance on a job or other measures of job success.

Compatibility of the Standards and the Principles with the Uniform Guidelines

The Uniform Guidelines were intended to be consistent with generally accepted professional standards for validating and evaluating standardized tests and other selection procedures. In this regard, the Guidelines specifically refer to the Standards.

It is strongly encouraged that you develop familiarity with both the Standards and the Principles in addition to the Uniform Guidelines. Together, they can help you conduct personnel assessment in a manner consistent with legal and professional standards.

CHAPTER 3 Understanding Test Quality—Concepts of Reliability and Validity

Test reliability and validity are two technical properties of a test that indicate the quality and usefulness of the test. These are the two most important features of a test. You should examine these features when evaluating the suitability of the test for your use. This chapter provides a simplified explanation of these two complex ideas. These explanations will help you to understand reliability and validity information reported in test manuals and reviews and use that information to evaluate the suitability of a test for your use.

Chapter Highlights

1. 1. What makes a good test?

2. 2. Test reliability

3. 3. Interpretation of reliability information from test manuals and reviews

4. 4. Types of reliability estimates

5. 5. Standard error of measurement

6. 6. Test validity

7. 7. Methods for conducting validation studies

8. 8. Using validity evidence from outside studies

9. 9. How to interpret validity information from test manuals and independent reviews.

Principles of Assessment Discussed

Use only reliable assessment instruments and procedures.

Use only assessment procedures and instruments that have been demonstrated to be valid for the
specific purpose for which they are being used.
Use assessment tools that are appropriate for the target population.

1. What makes a good test?

An employment test is considered “good” if the following can be said about it:

� The test measures what it claims to measure consistently or reliably. This means that if a person were to take the test again, the person would get a similar test score.

� The test measures what it claims to measure. For example, a test of mental ability does in fact measure mental ability, and not some other characteristic.

� The test is job-relevant. In other words, the test measures one or more characteristics that are important to the job.

� By using the test, more effective employment decisions can be made about individuals. For example, an arithmetic test may help you to select qualified workers for a job that requires knowledge of arithmetic operations.

The degree to which a test has these qualities is indicated by two technical properties: reliability and validity.

2. Test reliability

Reliability refers to how dependably or consistently a test measures a characteristic. If a person takes the test again, will he or she get a similar test score, or a much different score? A test that yields similar scores for a person who repeats the test is said to measure a characteristic reliably.

How do we account for an individual who does not get exactly the same test score every time he or she takes the test? Some possible reasons are the following:

� Test taker’s temporary psychological or physical state. Test performance can be influenced by a person’s psychological or physical state at the time of testing. For example, differing levels of anxiety, fatigue, or motivation may affect the applicant’s test results.

� Environmental factors. Differences in the testing environment, such as room temperature, lighting, noise, or even the test administrator, can influence an individual’s test performance.

� Test form. Many tests have more than one version or form. Items differ on each form, but each form is supposed to measure the same thing. Different forms of a test are known as parallel forms or alternate forms. These forms are designed to have similar measurement characteristics, but they contain different items. Because the forms are not exactly the same, a test taker might do better on one form than on another.

� Multiple raters. In certain tests, scoring is determined by a rater’s judgments of the test taker’s performance or responses. Differences in training, experience, and frame of reference among raters can produce different test scores for the test taker.

These factors are sources of chance or random measurement error in the assessment process. If there were no random errors of measurement, the individual would get the same test score, the individual’s “true” score, each time. The degree to which test scores are unaffected by measurement errors is an indication of the reliability of the test.

Reliable assessment tools produce dependable, repeatable, and consistent information about people. In order to meaningfully interpret test scores and make useful employment or career-related decisions, you need reliable tools. This brings us to the next principle of assessment.

Principle of Assessment

Use only reliable assessment instruments and procedures. In other words, use only assessment tools that provide dependable and consistent information.

3. Interpretation of reliability information from test manuals and reviews

Test manuals and independent review of tests provide information on test reliability. The following discussion will help you interpret the reliability information about any test.

The reliability of a test is indicated by the reliability coefficient. It is denoted by the letter “r,” and is expressed as a number ranging between 0 and 1.00, with r = 0 indicating no reliability, and r = 1.00 indicating perfect reliability. Do not expect to find a test with perfect reliability. Generally, you will see the reliability of a test as a decimal, for example, r = .80 or r = .93. The larger the reliability coefficient, the more repeatable or reliable the test scores. Table 1 serves as a general guideline for interpreting test reliability. However, do not select or reject a test solely based on the size of its reliability coefficient. To evaluate a test’s

Table 1. General Guidelines for Interpreting Reliability Coefficients

Reliability
coefficient
value	Interpretation
.90 and up	excellent
.80 - .89	good
.70 - .79	adequate
below .70	may have limited
	applicability

reliability, you should consider the type of test, the type of reliability estimate reported, and the context in which the test will be used.

4. Types of reliability estimates

There are several types of reliability estimates, each influenced by different sources of measurement error. Test developers have the responsibility of reporting the reliability estimates that are relevant for a particular test. Before deciding to use a test, read the test manual and any independent reviews to determine if its reliability is acceptable. The acceptable level of reliability will differ depending on the type of test and the reliability estimate used.

The discussion in Table 2 should help you develop some familiarity with the different kinds of reliability estimates reported in test manuals and reviews.

Table 2. Types of Reliability Estimates

# Test-retest reliability indicates the repeatability of test scores with the passage of time. This estimate also reflects the stability of the characteristic or construct being measured by the test.

Some constructs are more stable than others. For example, an individual’s reading ability is more stable over a particular period of time than that individual’s anxiety level. Therefore, you would expect a higher test-retest reliability coefficient on a reading test than you would on a test that

measures anxiety. For constructs that are expected to vary over time, an acceptable test-retest reliability coefficient may be lower than is suggested in Table 1.

# Alternate or parallel form reliability indicates how consistent test scores are likely to be if a person takes two or more forms of a test.

A high parallel form reliability coefficient indicates that the different forms of the test are very similar which means that it makes virtually no difference which version of the test a person takes. On the other hand, a low parallel form reliability coefficient suggests that the different forms are probably not comparable; they may be measuring different things and therefore cannot be used interchangeably.

# Inter-rater reliability indicates how consistent test scores are likely to be if the test is scored by two or more raters.

On some tests, raters evaluate responses to questions and determine the score. Differences in judgments among raters are likely to produce variations in test scores. A high inter-rater reliability coefficient indicates that the judgment process is stable and the resulting scores are reliable.

Inter-rater reliability coefficients are typically lower than other types of reliability estimates. However, it is possible to obtain higher levels of inter-rater reliabilities if raters are appropriately trained.

# Internal consistency reliability indicates the extent to which items on a test measure the same thing.

A high internal consistency reliability coefficient for a test indicates that the items on the test are very similar to each other in content (homogeneous). It is important to note that the length of a test can affect internal consistency reliability. For example, a very lengthy test can spuriously inflate the reliability coefficient.

Tests that measure multiple characteristics are usually divided into distinct components. Manuals for such tests typically report a separate internal consistency reliability coefficient for each component in addition to one for the whole test.

Test manuals and reviews report several kinds of internal consistency reliability estimates. Each type of estimate is appropriate under certain circumstances. The test manual should explain why a particular estimate is reported.

5. Standard error of measurement

Test manuals report a statistic called the standard error of measurement (SEM). It gives the margin of error that you should expect in an individual test score because of imperfect reliability of the test. The SEM represents the degree of confidence that a person’s “true” score lies within a particular range of scores. For example, an SEM of “2" indicates that a test taker’s “true” score probably lies within 2 points in either direction of the score he or she receives on the test. This means that if an individual receives a 91 on the test, there is a good chance that the person’s “true” score lies somewhere between 89 and 93.

The SEM is a useful measure of the accuracy of individual test scores. The smaller the SEM, the more accurate the measurements.

When evaluating the reliability coefficients of a test, it is important to review the explanations provided in the manual for the following:

� Types of reliability used. The manual should indicate why a certain type of reliability coefficient was reported. The manual should also discuss sources of random measurement error that are relevant for the test.

� How reliability studies were conducted. The manual should indicate the conditions under which the data were obtained, such as the length of time that passed between administrations of a test in a test-retest reliability study. In general, reliabilities tend to drop as the time between test administrations increases.

� The characteristics of the sample group. The manual should indicate the important characteristics of the group used in gathering reliability information, such as education level, occupation, etc. This will allow you to compare the characteristics of the people you want to test with the sample group. If they are sufficiently similar, then the reported reliability estimates will probably hold true for your population as well.

For more information on reliability, consult the APA Standards, the SIOP Principles, or any major textbook on psychometrics or employment testing. Appendix A lists some possible sources.

6. Test validity

Validity is the most important issue in selecting a test. Validity refers to what characteristic the test measures and how well the test measures that characteristic.

� Validity tells you if the characteristic being measured by a test is related to job qualifications and requirements. � Validity gives meaning to the test scores. Validity evidence indicates that there is linkage between test performance and job performance. It can tell you what you may conclude or predict about

someone from his or her score on the test. If a test has been demonstrated to be a valid predictor

of performance on a specific job, you can conclude that persons scoring high on the test are more

likely to perform well on the job than persons who score low on the test, all else being equal.

� Validity also describes the degree to which you can make specific conclusions or predictions about people based on their test scores. In other words, it indicates the usefulness of the test.

It is important to understand the differences between reliability and validity. Validity will tell you how good a test is for a particular situation; reliability will tell you how trustworthy a score on that test will be. You cannot draw valid conclusions from a test score unless you are sure that the test is reliable. Even when a test is reliable, it may not be valid. You should be careful that any test you select is both reliable and valid for your situation.

A test’s validity is established in reference to a specific purpose; the test may not be valid for different purposes. For example, the test you use to make valid predictions about someone’s technical proficiency on the job may not be valid for predicting his or her leadership skills or absenteeism rate. This leads to the next principle of assessment.

Principle of Assessment

Use only assessment procedures and instruments that have been demonstrated to be valid for the specific purpose for which they are being used.

Similarly, a test’s validity is established in reference to specific groups. These groups are called the reference groups. The test may not be valid for different groups. For example, a test designed to predict the performance of managers in situations requiring problem solving may not allow you to make valid or meaningful predictions about the performance of clerical employees. If, for example, the kind of problem-solving ability required for the two positions is different, or the reading level of the test is not suitable for clerical applicants, the test results may be valid for managers, but not for clerical employees.

Test developers have the responsibility of describing the reference groups used to develop the test. The manual should describe the groups for whom the test is valid, and the interpretation of scores for individuals belonging to each of these groups. You must determine if the test can be used appropriately with the particular type of people you want to test. This group of people is called your target population or target group.

Principle of Assessment

Use assessment tools that are appropriate for the target population.

Your target group and the reference group do not have to match on all factors; they must be sufficiently similar so that the test will yield meaningful scores for your group. For example, a writing ability test developed for use with college seniors may be appropriate for measuring the writing ability of white-collar professionals or managers, even though these groups do not have identical characteristics. In determining the appropriateness of a test for your target groups, consider factors such as occupation, reading level, cultural differences, and language barriers.

Recall that the Uniform Guidelines require assessment tools to have adequate supporting evidence for the conclusions you reach with them in the event adverse impact occurs. A valid personnel tool is one that measures an important characteristic of the job you are interested in. Use of valid tools will, on average, enable you to make better employment-related decisions. Both from business-efficiency and legal viewpoints, it is essential to only use tests that are valid for your intended use.

In order to be certain an employment test is useful and valid, evidence must be collected relating the test to a job. The process of establishing the job relatedness of a test is called validation.

7. Methods for conducting validation studies

The Uniform Guidelines discuss the following three methods of conducting validation studies. The Guidelines describe conditions under which each type of validation strategy is appropriate. They do not express a preference for any one strategy to demonstrate the job-relatedness of a test.

� Criterion-related validation requires demonstration of a correlation or other statistical relationship between test performance and job performance. In other words, individuals who score high on the test tend to perform better on the job than those who score low on the test. If the criterion is obtained at the same time the test is given, it is called concurrent validity; if the criterion is obtained at a later time, it is called predictive validity.

� Content-related validation requires a demonstration that the content of the test represents important job-related behaviors. In other words, test items should be relevant to and measure directly important requirements and qualifications for the job.

� Construct-related validation requires a demonstration that the test measures the construct or characteristic it claims to measure, and that this characteristic is important to successful performance on the job.³

The three methods of validity—criterion-related, content, and construct—should be used to provide validation support depending on the situation. These three general methods often overlap, and, depending on the situation, one or more may be appropriate. French (1990) offers situational examples of when each method of validity may be applied.

³Current thinking in psychology is that construct validity encompasses all other forms of validity; validation is the cumulative and on-going process of giving meaning to test scores.

First, as an example of criterion-related validity, take the position of millwright. Employees’ scores (predictors) on a test designed to measure mechanical skill could be correlated with their performance in servicing machines (criterion) in the mill. If the correlation is high, it can be said that the test has a high degree of validation support, and its use as a selection tool would be appropriate.

Second, the content validation method may be used when you want to determine if there is a relationship between behaviors measured by a test and behaviors involved in the job. For example, a typing test would be high validation support for a secretarial position, assuming much typing is required each day. If, however, the job required only minimal typing, then the same test would have little content validity. Content validity does not apply to tests measuring learning ability or general problem-solving skills (French, 1990).

Finally, the third method is construct validity. This method often pertains to tests that may measure abstract traits of an applicant. For example, construct validity may be used when a bank desires to test its applicants for “numerical aptitude.” In this case, an aptitude is not an observable behavior, but a concept created to explain possible future behaviors. To demonstrate that the test possesses construct validation support, “. . . the bank would need to show (1) that the test did indeed measure the desired trait and (2) that this trait corresponded to success on the job” (French, 1990, p. 260).

Professionally developed tests should come with reports on validity evidence, including detailed explanations of how validation studies were conducted. If you develop your own tests or procedures, you will need to conduct your own validation studies. As the test user, you have the ultimate responsibility for making sure that validity evidence exists for the conclusions you reach using the tests. This applies to all tests and procedures you use, whether they have been bought off-the-shelf, developed externally, or developed in-house.

Validity evidence is especially critical for tests that have adverse impact. When a test has adverse impact, the Uniform Guidelines require that validity evidence for that specific employment decision be provided.

The particular job for which a test is selected should be very similar to the job for which the test was originally developed. Determining the degree of similarity will require a job analysis. Job analysis is a systematic process used to identify the tasks, duties, responsibilities and working conditions associated with a job and the knowledge, skills, abilities, and other characteristics required to perform that job.

Job analysis information may be gathered by direct observation of people currently in the job, interviews with experienced supervisors and job incumbents, questionnaires, personnel and equipment records, and work manuals. In order to meet the requirements of the Uniform Guidelines, it is advisable that the job analysis be conducted by a qualified professional, for example, an industrial and organizational psychologist or other professional well trained in job analysis techniques. Job analysis information is central in deciding what to test for and which tests to use.

TESTING AND ASSESSMENT: AN EMPLOYER’S GUIDE TO GOOD PRACTICES

TESTING AND ASSESSMENT:
AN EMPLOYER’S GUIDE
TO GOOD PRACTICES