Seven Questions to Ask a Vendor Before Purchasing Employee Assessments
As a hiring manager, employee testing has become a regular, but not always well executed, part of the talent management process.
However, if you have access to best-in-class tools and professional cost-effective interpretation support, the employee testing experience and the results are worth their weight in gold.
We will assume that you have identified a set of competencies and the requisite knowledge, skills, and abilities required.
Now what? How do you find the best-in-class testing program that can be customized to meet your specific needs AND provide you with concrete, easily understood information from which you can make objective and sound decisions? To add a dimension of complexity to the situation managers typically do not have the luxury of time, resources, or background to engage in test development and validation.
Because of this reality we have developed this information to assist managers in not only planning and executing proper employee testing programs but also to be able to objectively evaluate and review the work being offered by consultants selling to them.
In making a proper decision related to employee testing programs you must ask the following seven questions before making a purchase.
If the seller cannot or will not answer the questions it would be wise to move on to another provider.
Any professional, knowledgeable and trained assessment advisor will welcome a client's detailed interest in their products and services.
They also want to make sure clients/organizations are getting the best solution for their specific needs regardless of whether it entails their own product line or not.
Now let's move on to the essential questions.
1.
What specifically does the test measure? While seemingly obvious, listen for a clear and concise answer to this question.
It is impossible to develop a good quality test without a clear definition of what is to be measured.
Keep in mind how the answer compares with what you need to be assessed in your organization.
Be cautious of any vendor who describes a single test as being able to determine or measure many seemingly unrelated factors.
This is a distraction.
No test can do everything yet many vendors try to pawn off their one test as the single choice for all outcomes.
Also, pay attention to any vendor who tries to sell the benefits of a tests that is short (i.
e.
has a limited number of test items).
For example a 20-item test that purports to measure 8 different personality factors.
This is simply problematic.
Measuring a personality traits or cluster of traits accurately typically requires more than just a couple of items.
2.
What research and process was used to develop the test? This is core to determining the utility of the test.
Find out what theory or methodology was used to develop the test? Was the test developed on people that are similar to your organizations applicants or employees? What was the process used to develop the test? At the very least, the vendor should be able to provide validation studies to support the uses of the test.
If validation studies are mentioned keep in mind that there are three types of validation.
See below for more details on this topic.
3.
What hands-on experience and professional education does the vendor have that qualifies them to use, sell or advise with the test? The educational background and work experience of the persons who developed or sells the test is critical.
As well, references that can speak to the capabilities and experience of the test vendor would be highly recommended.
To have confidence in the test and in the event of a legal challenge, you want test developers and their respective vendors to have education and/or experience related to the specific content of the test and related to test development and validation.
Also, some tests require the test administrators or individuals interpreting test scores have certain credentials (e.
g.
, MA, PhD) that reflect coursework in statistics, test interpretation, or test development and validation.
The internet has unfortunately opened the floodgate to unregulated individuals presenting themselves as certified experts when in fact a deeper look at credentials often highlights limited and narrowly defined training, typically in the one test they happen to be selling.
4.
What research does the vendor have that is related to the reliability of this test? In the most simple of terms, reliability refers the consistency of a tests result.
There are several ways to assess the reliability of a test.
Some are more appropriate for certain situations (e.
g.
, when multiple raters or evaluators are involved; if one wishes to know about stability of results over time).
Experienced and knowledgeable test publishers and vendors will have information on the reliability of their testing products.
This is typically provided in a technical manual which outlines in detail the procedures and protocols which were used in developing the test.
5.
What research evidence does the vendor have related to the validity of this test? Validity refers to the accuracy of the inferences made based on test results (e.
g.
, how accurate is it to say that a higher test score indicates that a person is more likely to be a better performer).
Knowledgeable and experienced test publishers typically have many forms of validity evidence.
For example, they may have evidence that shows a relationship between test scores and some outcome of interest (e.
g.
, supervisory ratings of job performance, average monthly sales, turnover).
They might also have evidence that documents a link between the content of the test and the requirements of the job.
Other evidence might include showing how the test relates to other measures of the same thing.
Experienced and knowledgeable test publishers have (and are happy to provide) information on the validity of their testing products.
Judgments regarding what types of validity evidence are appropriate for a given test depend on a number of factors, and these are outlined in The Standards for Educational and Psychological Testing (www.
apa.
org/science/standards.
html ) For more detailed information on how to assess the validity evidence associated with a particular test, check out Testing and Assessment: An Employers Guide to Good Practices at www.
onetcenter.
org.
6.
What evidence do you have that demonstrates the lack of bias or discrimination of your test? Look for evidence that the test does not contain bias on the basis of race or sex, that is, that the test is related to outcomes in a similar manner for all individuals.
This statement does not necessarily mean that the test will have similar results for different groups of people.
This statement does mean that the test is not a biased indicator of an outcome of interest.
For example, in a typical employment decision context, more women than men will score low on a test of upper body strength.
The test, however, would not be considered biased if women and men with similar scores achieved similar performance on the job.
7.
What data do you have that will help me interpret test scores in my organization? You cannot interpret test scores by themselves.
Whether a test score is considered good or poor may depend on the distribution of scores of a comparison group.
This comparison group is typically referred to as a norm group.
The test publisher should provide information about the different norm groups that are available for the test being considered.
Ideally, you want to use a norm group that is similar to the group of people that are in the position for which testing is being used.
There are other ways to interpret test results including expectancy charts and cut scores, which are developed based on information about how the score relates to outcomes of interest.
Information should be made available on data that can aid in appropriate test score interpretation.
To learn more about how we can assist you in developing best-in-class, standardized and validated evidence-based employee testing programs give me a call on my personal line at 416-561-6817.
I would be pleased to learn more about your needs and how we can best support your testing objectives.
Dr.
Richard
However, if you have access to best-in-class tools and professional cost-effective interpretation support, the employee testing experience and the results are worth their weight in gold.
We will assume that you have identified a set of competencies and the requisite knowledge, skills, and abilities required.
Now what? How do you find the best-in-class testing program that can be customized to meet your specific needs AND provide you with concrete, easily understood information from which you can make objective and sound decisions? To add a dimension of complexity to the situation managers typically do not have the luxury of time, resources, or background to engage in test development and validation.
Because of this reality we have developed this information to assist managers in not only planning and executing proper employee testing programs but also to be able to objectively evaluate and review the work being offered by consultants selling to them.
In making a proper decision related to employee testing programs you must ask the following seven questions before making a purchase.
If the seller cannot or will not answer the questions it would be wise to move on to another provider.
Any professional, knowledgeable and trained assessment advisor will welcome a client's detailed interest in their products and services.
They also want to make sure clients/organizations are getting the best solution for their specific needs regardless of whether it entails their own product line or not.
Now let's move on to the essential questions.
1.
What specifically does the test measure? While seemingly obvious, listen for a clear and concise answer to this question.
It is impossible to develop a good quality test without a clear definition of what is to be measured.
Keep in mind how the answer compares with what you need to be assessed in your organization.
Be cautious of any vendor who describes a single test as being able to determine or measure many seemingly unrelated factors.
This is a distraction.
No test can do everything yet many vendors try to pawn off their one test as the single choice for all outcomes.
Also, pay attention to any vendor who tries to sell the benefits of a tests that is short (i.
e.
has a limited number of test items).
For example a 20-item test that purports to measure 8 different personality factors.
This is simply problematic.
Measuring a personality traits or cluster of traits accurately typically requires more than just a couple of items.
2.
What research and process was used to develop the test? This is core to determining the utility of the test.
Find out what theory or methodology was used to develop the test? Was the test developed on people that are similar to your organizations applicants or employees? What was the process used to develop the test? At the very least, the vendor should be able to provide validation studies to support the uses of the test.
If validation studies are mentioned keep in mind that there are three types of validation.
See below for more details on this topic.
3.
What hands-on experience and professional education does the vendor have that qualifies them to use, sell or advise with the test? The educational background and work experience of the persons who developed or sells the test is critical.
As well, references that can speak to the capabilities and experience of the test vendor would be highly recommended.
To have confidence in the test and in the event of a legal challenge, you want test developers and their respective vendors to have education and/or experience related to the specific content of the test and related to test development and validation.
Also, some tests require the test administrators or individuals interpreting test scores have certain credentials (e.
g.
, MA, PhD) that reflect coursework in statistics, test interpretation, or test development and validation.
The internet has unfortunately opened the floodgate to unregulated individuals presenting themselves as certified experts when in fact a deeper look at credentials often highlights limited and narrowly defined training, typically in the one test they happen to be selling.
4.
What research does the vendor have that is related to the reliability of this test? In the most simple of terms, reliability refers the consistency of a tests result.
There are several ways to assess the reliability of a test.
Some are more appropriate for certain situations (e.
g.
, when multiple raters or evaluators are involved; if one wishes to know about stability of results over time).
Experienced and knowledgeable test publishers and vendors will have information on the reliability of their testing products.
This is typically provided in a technical manual which outlines in detail the procedures and protocols which were used in developing the test.
5.
What research evidence does the vendor have related to the validity of this test? Validity refers to the accuracy of the inferences made based on test results (e.
g.
, how accurate is it to say that a higher test score indicates that a person is more likely to be a better performer).
Knowledgeable and experienced test publishers typically have many forms of validity evidence.
For example, they may have evidence that shows a relationship between test scores and some outcome of interest (e.
g.
, supervisory ratings of job performance, average monthly sales, turnover).
They might also have evidence that documents a link between the content of the test and the requirements of the job.
Other evidence might include showing how the test relates to other measures of the same thing.
Experienced and knowledgeable test publishers have (and are happy to provide) information on the validity of their testing products.
Judgments regarding what types of validity evidence are appropriate for a given test depend on a number of factors, and these are outlined in The Standards for Educational and Psychological Testing (www.
apa.
org/science/standards.
html ) For more detailed information on how to assess the validity evidence associated with a particular test, check out Testing and Assessment: An Employers Guide to Good Practices at www.
onetcenter.
org.
6.
What evidence do you have that demonstrates the lack of bias or discrimination of your test? Look for evidence that the test does not contain bias on the basis of race or sex, that is, that the test is related to outcomes in a similar manner for all individuals.
This statement does not necessarily mean that the test will have similar results for different groups of people.
This statement does mean that the test is not a biased indicator of an outcome of interest.
For example, in a typical employment decision context, more women than men will score low on a test of upper body strength.
The test, however, would not be considered biased if women and men with similar scores achieved similar performance on the job.
7.
What data do you have that will help me interpret test scores in my organization? You cannot interpret test scores by themselves.
Whether a test score is considered good or poor may depend on the distribution of scores of a comparison group.
This comparison group is typically referred to as a norm group.
The test publisher should provide information about the different norm groups that are available for the test being considered.
Ideally, you want to use a norm group that is similar to the group of people that are in the position for which testing is being used.
There are other ways to interpret test results including expectancy charts and cut scores, which are developed based on information about how the score relates to outcomes of interest.
Information should be made available on data that can aid in appropriate test score interpretation.
To learn more about how we can assist you in developing best-in-class, standardized and validated evidence-based employee testing programs give me a call on my personal line at 416-561-6817.
I would be pleased to learn more about your needs and how we can best support your testing objectives.
Dr.
Richard