VALIDATION

Standards

So states the Committee on Ability Testing of the National Research Council as quoted from Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 1985 edition, p.1). (Standards will be used to refer to that text henceforth.) That publication also points out that not all tests are well-developed, nor all testing practices wise and beneficial. Therefore, establishing the validity and reliability of the testing procedure -- that is, both the test instrument and test administration -- is an important first step before the test can be employed with confidence.

Validity & Reliability

Without first-hand experience with the predictive ability of a test, external measures of validity and reliability help the decision maker to choose between alternative tests. The term validity is used to convey the idea that the test procedure measures what it purports to measure. For example, a ruler may be employed in a more valid measurement of length than a thermometer. Face validity, a measurement of validity based upon readily accepted judgments, is a property of many inventories which ask questions that are directly related to the trait being measured. For example, for the measurement of the trait ‘sociability,’ the question “Do you enjoy meeting new people,” would be generally accepted as face valid. This approach was taken in the development of the REMBRANDT PORTRAIT^®.

Another factor that affects validity is respondent biases. Asking questions about traits directly (such as, “How social are you?”) can elicit consciously or unconsciously biased responses.

Instead, inventories such as the REMBRANDT PORTRAIT^® are employed to estimate personality variables by examining a set of responses to specific statements consonant with the trait. For example, instead of asking “How social are you?”, the statements might include: “I make friends easily,” “I enjoy a lot of company,” and so forth. These statements are presented as alternatives to other positive-quality statements, such as “I do my work responsibly,” “I do not tire easily,” and the like. Therefore, in choosing one positive trait over another, it is very difficult for an individual to bias the results in favor of a particular quality without depreciating another positive quality. The REMBRANDT PORTRAIT^® makes use of this method, selecting from among competing positive traits, to minimize respondent biases.

The other concern of the decision maker, reliability, also bears upon the utility of an inventory. Reliability of a test concerns the degree of test-measurement error. A common measurement of reliability is the consistency between two administrations of the inventory to the same people. Factors contributing to low reliability in tests include, but are not limited to the following: (1) differences in the way the test instrument was administered, (2) interactions between the test giver and the test taker, (3) cultural differences, (4) differences in the evaluations and interpretations of test scoring.

Choosing an inventory

In deciding to use any particular test to aid in a decision process, the decision maker must either be familiar with the use of the test or be familiar with the standards to which the test was validated. Consistent with Standards, we note that it is not appropriate for test developers or users to state that a test satisfies or follows the standards set forth by that publication. Rather, based upon both the validation data available and the experience with the test, professional judgments must be made.

Choosing an inventory

The REMBRANDT PORTRAIT^® is a computer-based test of fourteen personality traits which are used for evaluating candidates for employment. Because it is administered by a PC-computer rather than by different administrators, variability both within and between administrators, which might detrimentally affect reliability and validity, is eliminated. Therefore, by employing a computer to administer the test, it may be more reliable and, therefore, possibly more valid than it would be otherwise. This document provides an overall examination of the REMBRANDT PORTRAIT^®, administered to persons already employed, and compares their REMBRANDT PORTRAIT^® -scores with self evaluations given by means of a standardly administered inventory. By comparing REMBRANDT PORTRAIT^® scores with face-valid inventory scores in persons who have no reason to conceal their true personality traits, the REMBRANDT PORTRAIT^® creators tried to obtain data that would demonstrate validity by reference to the inventory scores. To compare the REMBRANDT PORTRAIT^® results to standardly administered inventory results, correlations were computed for each of the ten primary items of the REMBRANDT PORTRAIT^®. In addition, to demonstrate the extent to which the ten REMBRANDT PORTRAIT^® traits reflected separate, distinct measures of personality, a factor analysis was employed.

Analysis

The ten primary traits of the REMBRANDT PORTRAIT© correspond to the following characteristics:

TRAIT -- CHARACTERISTICS

1. Aggressiveness -- willing to criticize, confrontative
2. Assertiveness -- leader, controls situation, less self-doubts
3. Caution (approach to decisions) -- minimizes risks, thoughtful decision maker
4. Correctness (sensitivity to rules / guidelines) -- follows rules, does things the proper way
5. Energy (stamina) -- works hard, lots of energy
6. Friendliness (sociability) -- enjoys making friends, likes to mix socially
7. Nurturance (service motivation) -- helpful, understanding
8. Pace / Urgency -- works faster, intense, impatient
9. Responsibility -- reliable, persistent, thorough
10. Structure (self-discipline) -- orderly, plans things, sticks to plan

NeuroCommunication Research Laboratories, Inc. (NCRL) analyzed personality-trait components of the REMBRANDT PORTRAIT^® in order to obtain information about the REMBRANDT PORTRAIT^® as an employment assessment tool. NCRL was provided with data collected during 1989 / 1990 from 143 employed individuals who completed both the REMBRANDT PORTRAIT^® inventory and also a traditionally administered, “paper-and-pencil” personality inventory (referred to as Inventory henceforth). Paired data were provided on each of the 10 primary personality traits. The study population was derived from a variety of industries, job-levels, and educational levels. There was a mix of age-levels, sexes, and racial groups.

The REMBRANDT PORTRAIT^® data were taken directly from the REMBRANDT PORTRAIT^® computer program, which assigned numerical values to statements selected by respondents interacting with the REMBRANDT PORTRAIT^® computer program. To determine each personality trait, multiple statements were utilized. The REMBRANDT PORTRAIT^® computer program categorized responses according the personality trait that they reflected. The sum of the numerical values for each personality trait was the raw score for that trait.

The Inventory presented statements describing hypothetical individuals who would be considered to have a high degree or a low degree of a personality trait. Respondents were asked to highlight the area of an unnumbered, 50-division scale which corresponded to their self-perception on the particular trait. Data were converted by linear transformation into a 0100 scale. The center of the highlighted area was used in the analysis to represent self perception.

Pearson Product-Moment Correlations were computed between the paired personality traits from the REMBRANDT PORTRAIT^® and Inventory. The correlation coefficients for each trait follow.

Each of the correlation coefficients is statistically significantly positive at better than the 99% confidence level (p<.001, two-tail).

The following conclusion can be drawn from these analyses:

The computer-based administration method of the REMBRANDT PORTRAIT^® provides a personality inventory that was generally comparable to a traditionally administered “paper-and-pencil” inventory in a sample of employed persons.

Factors

In order to determine whether the REMBRANDT PORTRAIT^® traits measure distinct constructs, NCRL performed a factor analysis on the REMBRANDT PORTRAIT^® personality traits. The procedure employed the method of Principal Components with Varimax rotation. To allow full freedom of effect, no restrictions were placed on the components before rotation. The results of this analysis yielded 9 factors (i.e., components) which accounted for approximately 99% of the variance. Eight of the factors corresponded specifically to a personality trait, that is, the factors loaded highly on one trait (.88.99) and low on all of the other traits (<.49). The percentage of total variance explained by each of the trait-related factors is presented below:

The other factor loaded highly on two of the remaining traits.

Correctness / Aggressiveness -- 14.87

A similar factor analysis was computed on the Inventory. The varimax rotation yield 10 factors yielding approximately 10% explained variance each (range: 9.9-10.1). Each factor loaded highly (.93 or above) on only one personality trait and low (.18 or below) on all other traits.

The following general conclusion can be drawn from these factor analyses:

Strength of Single Factored Constructs

This result (8 single-factored constructs) shows a strength Factored Constructs of the REMBRANDT PORTRAIT^®, yet this result contrasts with other results found for traditionally employed tests, many of which are multifactorial. Constructs which are multifactorial are undesirable, because of two pitfalls. The two pitfalls are: (1) Internal consistency reliability of the overall construct is weakened; (2) The degree of validity can be compromised.

Unlike some other employment inventories, most of the traits on the REMBRANDT PORTRAIT^® are single factored. The single multifactorial construct of the REMBRANDT PORTRAIT^® is two factored (combining correctness and aggressiveness), which would allow deviations in the anticipated association (i.e., more aggressive is less correct on details) to be separated by subsequent interviewing.

REMBRANDT PORTRAIT^® Consistency With Other Inventories

There does seem to be a strong correspondence of the traits measured in the REMBRANDT PORTRAIT^® and traits measured in other established personality inventories. For illustrate purposes, REMBRANDT PORTRAIT^® traits are listed which generally correspond with two better known personality inventories, The Personality Research Form (PRF) and the Edwards Personality Inventory (EPI).

These inventories were developed to provide an instrument for measuring broadly relevant personality traits for use in education, business, and counseling, a goal consistent with that of the REMBRANDT PORTRAIT^®.

The PRF has 20 traits of which 9 correspond with the 10 primary traits of the REMBRANDT PORTRAIT^®. The corresponding traits of the PRF are further described in the following table. These descriptive adjectives are the ones presented in the PRF manual.

The EPI has over 50 scales, several of which relate to a common personality construct. All of the 10 primary traits of the REMBRANDT PORTRAIT^® relate to one or several of the EPI scales.

Table 1. REMBRANDT PORTRAIT^® and PRF Adjectives

Table 2. REMBRANDT PORTRAIT^® and EPI Adjectives

Other personality inventories, such as the Gordon Personal Profile and Cattell’s Sixteen Personality Factors Questionnaire (16 PF) also have traits which generally correspond with the REMBRANDT PORTRAIT^®.

Summary

In summary, the REMBRANDT PORTRAIT^® is a rapid personality assessment tool, which yields ten primary measurement scales. The REMBRANDT PORTRAIT^® is objectively administered and scored by computer. Potential employers can personalize the test for their particular situation, instead of applying one set of norms for all. This analysis indicates that there is very little overlap in what the ten primary scales measure. Additionally, the REMBRANDT PORTRAIT^® traits correspond with a traditionally administered “paper-and-pencil” inventory. The traits measured by the REMBRANDT PORTRAIT^® are likely to be useful for screening for all employment situations.

NOTE

NeuroCommunication Research Laboratories (NCRL), an independent laboratory, functioned as a consultant to the creators of the REMBRANDT PORTRAIT^® to prepare this document. Though not directly involved with the creation of the REMBRANDT PORTRAIT^®, the study design, the data collection, the preliminary data reduction, or the data entry, NCRL performed the statistical analyses and the interpretations that are presented in this document.

VALIDATION STUDY RESULTS CHART 12/29/90

Degree of match between self assessment questionnaire and REMBRANDT PORTRAIT^® results: 143 Individuals

CONCLUSION: About two-thirds of the traits measured are within 5%

About 85% of the traits measured are within 15%

Test / Re-test Reliability

Another phase of this study was to determine the tests’ test – re-test reliability. It is important for any employment measuring tool to consistently measure an individual over time. The 143 people involved in this study were administered the REMBRANDT PORTRAIT^® several times over a period of 18 months.

Findings:

Test / Re-test reliability coefficients:

Conclusion

The REMBRANDT PORTRAIT^® has strong internal stability resulting in strong correlation between the original test and subsequent tests.

REMBRANDT PORTRAIT^® Technical Manual

Part II

Part I of Rembrandt Portrait Technical Manual© outlines the Construct and Reliability study approaches and findings conducted by Sir Stanley Weinstien, Ph.D., Chief Scientist at the NeuroCommunication Research Laboratories, Inc.

This report concludes that Rembrandt Portrait© is a valid tool for measuring the personality traits it purports to measure, as compared to other established, valid and age proven personality inventories such as the Cattell 16 PF, The Edwards Inventory, The Gordon Profile and The Psychological Assessment. Rembrandt Portrait© scales are independent from one another, making The Rembrandt Portrait© a highly reliable, stable assessment.

The study also shows that the Rembrandt Portrait© is a highly reliable assessment tool, with test / retest variation over an 18 month time span of less than 8%, securing Rembrandt Portrait©’s place as one of the most reliable personality inventories created.

Rembrandt Portrait Technical Manual© Part II takes these study results to the next level, outlining three other forms of validation that should be of equal importance to any firm deciding to include a personality profile into its employee selection and / or development processes. Part II outlines study findings for Rembrandt Portrait©’s 1) predictive capability 2) impact on employee turnover, and 3) in-field impact on protected classes of employees.

As you will see from the following study outcomes Rembrandt Portrait© is highly predictive when used as a job matching tool, with recommended candidates outperforming non-recommended candidates at a better than 2.5 to 1 margin. This is outlined in exhibits one and two. Because Rembrandt Portrait© is so highly predictive of job success, it has proven capability to dramatically reduce selection-based employee turnover (turnover incurred in the first 6 months of employment). In study after study, Rembrandt Portrait© has been able to reduce selection-based turnover by more than 50% (see exhibit 3 & 4). Finally, we have outlined the impact that Rembrandt Portrait© has on protected classes of employees. In exhibit 5 you will see that Rembrandt Portrait© has no adverse impact on protected classes of employees and actually supports fair and equitable hiring practices by identifying the right person for the right position, regardless of age, sex, race, religion, national origin and sexual orientation.

Exhibit II

This study was conducted within the Northwest division of an insurance and financial services firm employing over 4,000 employees. This study tracks the performance of new employees for all positions over a 2 year period and quantifies employee performance by recommended group – those candidates identified as "job matched" and those identified as "not job matched."

When Rembrandt Portrait© identified the candidate as "job matched", when employed, 85.5% of these individuals were rated by their manager as either "meeting" or "exceeding" position requirements.

When the candidate was identified as "not job matched," if employed, only 37.2% of these individuals were rated by their managers as either "meeting" or "exceeding" job requirements: a nearly 2.5 to 1 advantage when job matched.

Moreover, of the 85.5% rated as "meeting" or "exceeding" job requirements (job matched), nearly half of this group was identified as actually "exceeding" job requirements. Of the 37.2% rated as "meeting" or "exceeding" job requirements (not job matched), only 2.9% of this group was rated as "exceeding" job requirements.

See the following table for a breakdown of these numbers.

The above graphs show empirically that:

• 85.5% of candidates recommended by Rembrandt Portrait© were rated as “meets” or “exceeds” position requirements compared to only 37.2% of not recommended candidates. Recommended candidates were nearly 2.5 times more likely of succeeding that were not recommended candidates
• 36.1% of recommended candidates performed at an “exceeds” position requirements level compared to only 2.9% of those not recommended. A staggering 97% of all superior-performing candidates were recommended
• only 14.5% of recommended candidates failed to perform to position requirements compared to 62.8% of those not recommended
• only 10.8% of recommended candidates turned over within 18 months of being employed compared to 51.4% of those not recommended

Exhibit III

While exhibits I and II clearly show the power of Rembrandt Portrait©’s ability to predict on the job performance it is equally important to determine the level of turnover when Rembrandt Portrait© is incorporated into a firm’s selection processes. The study outlined below clearly shows that when individuals that are “job matched” are hired, selection based turnover plummets. Simply by hiring the right person for the right position, first year employee turnover was reduced by nearly 50% in less than one year.

The study outlined below was conducted within a large national staffing service with over 300 offices throughout North America. To ensure that we were measuring the impact of Rembrandt Portrait© in reducing turnover, one region of this firm, covering 19 offices, did not participate in the use of Rembrandt Portrait© during the course of the study. This enabled us to have a control group against which we could compare Rembrandt Portrait©’s impact.

Over the span of one year, those offices utilizing Rembrandt Portrait© as a part of their new employee selection process cut first year turnover by half, while turnover in the control group increased by 2% during the same period of time.

There is no question that Rembrandt Portrait© is a powerful tool for abating and controlling selection based employee turnover.

Exhibit IV

At the same time as conducting the study outlined above in exhibit III we conducted a similar study with a national Call Center. Over the course of two years the Call Center participating in this study tracked employee turnover for the purposes of establishing a baseline against which we would compare Rembrandt Portrait©’s impact on turnover. For this two year period, first year turnover averaged 89% annually. This means for every 10 employees hired at the beginning of the year only 1 individual remained on the job at the end of one year. When Rembrandt Portrait© was added to this firm’s selection process, in just one year, first year employee turnover plummeted to 36%.

Rembrandt Portrait© was able to help this firm reduce employee turnover by 60% in just one year. In addition, 27% of all employees hired with the help of Rembrandt Portrait© performed above job requirements while only 1% of all employees hired without the help of Rembrandt Portrait© performed at this above par level.

Exhibit V

The last data we researched was Rembrandt Portrait©’s impact on protected classes of employees. To accomplish this, we took a historical look at the employee mix of a large division of a multi-national service firm. As you will see in the following table, not only is Rembrandt Portrait© blind to age, race, sex, national origin and sexual orientation, when included in a selection process, the selection of protected groups of candidates actually increased. We attribute this to Rembrandt Portrait©’s ability to match the person to the position, thereby enabling the hiring manager to look past non-valid hiring criteria and to focus solely on the candidate’s ability to perform the job.

In summation, Rembrandt Portrait© is a highly valid personality assessment profile that has no adverse impact on protected groups. It can predict success on the job with 85.5% accuracy and can identify Superior performing individuals with 97% accuracy. When Rembrandt Portrait© is incorporated into a firm’s selection processes it can actually assist that firm in meeting or exceeding all Fair Employment standards, practices and laws.

Rembrandt Portrait© is:

• Valid
• Reliable
• Constructed to measure what it claims to measure
• Predictive of job success
• Non-discriminatory
• Legally defensible

Rembrandt Portrait© is so highly valid that it is the only profile offered with a full "Hold Harmless" agreement to employer corporations.

© Rembrandt Advantage. All Rights Reserved.