Assessing the assessor- Evaluating personality assessment tools

Imagine someone walking up to a doctor with a new kind of device that claims to measure the body temperature. What if the manufacturer requested the doctor to endorse the new product? The doctor is the head of a hospital and her endorsement could mean opening up a new market for the manufacturer. How should the doctor go about taking that decision?  

The doctor would check it for the safety and reliability of the readings, and calibrate it against thermometers used by the hospital. The doctor could check her own temperature a few times over the hour to check if the reading is consistent. In short, one would take all measures to check the reliability and validity of a product or a tool against the accepted standards.

Passing the “Thermometer Test”

Today, many organizations use personality assessment tools to assess their employees. Talent acquisition, job fit, and coaching are all great cases for the use of personality assessment tools. After all, the leader’s personality is what gets reflected in his or her leadership style and can be a great source of understanding the organization’s culture. But before deciding to use a personality tool, organizations should create their own “thermometer test”. It is important to know that the test that one is using is capable of doing what it promises to do and have an expert psychometrician decide the “reliability and validity” of the tool.

Here is a set of elements that one can use to choose the right assessment tool:

Check for the science behind the test

Check if a sufficient number of top peer-reviewed journals (e.g., Journal of Applied Psychology, Personnel Psychology, Human Performance, International Journal of Selection and Assessment etc.) has reviewed the tool. Check if the test adheres to globally accepted standards of designing personality assessment tools and check for the reviews in Buros’ Mental Measurement Yearbook or the British Psychological Society (BPS) standards or the Psychological Testing Center. Do not get carried away by the names of client organizations provided by the test provider.

Personality assessment tests need to be tested and calibrated by experts before they are administered to the employees 

Determining what the test will be used for 

The tests can be used for making decisions on hiring, the potential for success in a new role in case of an internal move, or to provide a basis for coaching a leader. It is important to define the key demands on the role for which a person is being evaluated. Then decide if the test that is being chosen measures what is wanted. You cannot use the thermometer to check the weight of a patient (at least, not yet).

Membership of bodies that regulate statistical and ethical standards

Check if your test provider is a member of the Society of Industrial/Organizational Psychology (SIOP), or some other professional organization that mandates ethical and statistical guidelines for creating assessments. It is like checking if a surgeon has the requisite experience defined by a professional body regulating guidelines applicable to surgeons or a Chartered Accountant is a member of the Institute of CAs — it is a proxy for the person’s ability to advise.

Test manual’s standards 

The test manual should adhere to the standards described by a body (e.g. SIOP) that regulates standards in development, validation, and norms for the assessment. The test manuals and reports must follow these standards. If your mobile phone manufacturer provides details on the radiation levels, standards around display, battery life etc., so must the personality assessment tool.

The Job 

The test being used should be validated for a job similar to the one it is being tested for. Ask the personality assessment provider to produce a summary of validation results for the job. Just as what is needed to succeed in Sales differs from what is needed in an Executive role, the test must be a valid predictor of performance in the target job. Ask the provider to explain how the high and low score cut-offs have been arrived at. That would help in understanding the norms against which the results will be compared. In the thermometer test, it is like asking the range of temperature that would be considered normal. A temperature of 98.6 degrees Fahrenheit is cast in stone as a measure of normal temperature. A body temperature of 99.7 degrees F or higher measured using an oral thermometer is considered fever. 

Just as what is needed to succeed in Sales differs from what is needed in an Executive role, the personality assessment test must be a valid predictor of performance in the target job

Test the test!

Personality assessment tests need to be developed by skilled psychometricians. They need to be tested and calibrated by experts before they are administered to the employees of your organization. Your employees trust you to do the due diligence before their career choices get shaped by an assessment. Try the ‘thermometer test’ with your assessment provider and evaluate the science behind the instrument.