Dr. Joni Lakin
CogAT co-author Dr. Joni Lakin was honored to be asked to contribute to a special issue of the Journal of Advanced Academics to provide a perspective on economists Card and Giuliano’s 2015 paper, “Can universal screening increase the representation of low income and minority students in gifted education?” Dr. Lakin and Dr. Matthew McBee wrote independent reviews of this paper translated for the gifted research field.
In the original study, Card and Giuliano took advantage of a “natural experiment,” where they were able to compare program diversity in a school district that moved from an identification process initiated by a teacher or parent referral to a new process that began with every second-grade student completing a screening assessment. This is called universal screening. The researchers were interested in the proportion of historically underrepresented minorities (such as English learners, Hispanic students, and African-American students) identified with the new program.
Basics of Universal Screening
Universal screening is an identification practice where all students in a targeted grade are administered an initial screening instrument. Scoring at or above a pre-determined cut-score on the screener leads to further consideration for placement and/or services in a gifted and talented program, usually involving at least one additional placement or confirmation assessment. The alternative to universal screening is often a referral process where parents or teachers recommend students for screening (or testing) for gifted services. Some research has suggested that a referral-only process introduces bias into the identification process and may lead to less representative gifted programs.
The key finding of their study was that the universal screening system was more effective than the previous teacher and parent referral system in addressing the under-identification of African-American, Hispanic, female, low socioeconomic status, and English learner students. Another important finding was that using universal screening greatly increased the number of students referred overall in the first screening stage, and therefore, required the second stage placement test to be identified for services.
The following analyses address some of the logistical issues of how to select screening and placement tests effectively and what effect liberal (low) versus restrictive (high) screening test cut-scores will have on the number of students who meet the cut-score on the placement test.
The data used in these analyses were gathered in a large, diverse school district in the southwest U.S. as part of a large-scale study in 2009. All the students took the complete CogAT® Form 6 and had two years of achievement test data. Therefore, we have the data on the “placement” test for all students in the study, allowing us to estimate the impact of the screening procedure.
We used data for Grades 3 to 6 including the CogAT 6 and mathematics, reading, and writing scores from the state’s 2009 and 2010 achievement tests. The CogAT 6 VQ (Verbal + Quantitative) composite was used as the placement test score, which is an average of the V and Q battery scores. The Nonverbal (N) Battery was treated as a highly correlated screening battery. Here’s a breakdown of the initial screener tests considered:
What impact does the correlation between screening and placement tests have on identification errors?
Our first research question was: When considering all the students who scored above a cut-score on the placement test, what proportion was identified or missed by the screening test?
For these analyses, we consider two types of “errors” that can be made using the universal screening test. False positive errors mean that students meet the cutoff at the initial stage for the screener test, but do not meet the cutoff on the placement test. False negative errors mean that students do not meet the cutoff on the placement test, but would have scored above the cutoff on the placement test. In the case of a false negative, the student misses the opportunity to qualify for the program at the second step by being eliminated in the initial stage. This latter error is especially concerning.
Most false positives (FP) and false negatives (FN) came from Writing, which had the lowest correlation with the placement test. All the other tests, which had stronger relationships to the placement test, had substantially lower false positive rates—i.e., students who met the cut-score on the screener test, but did not meet the higher cut-score on the placement test. So we can conclude that higher correlations between the screening and the placement test will reduce the costs of excessive placement testing.
The shorter tests (N and FA) had larger false negative rates than the two full achievement batteries (math and reading). This meant that a lenient cut-score on an achievement test resulted in the fewest errors where students who would meet the cutoff on the placement test were excluded by the screening test.
What impact does the liberality of the cut-score have on identification errors?
Our second research question addressed whether setting a liberal (low) vs. a restrictive (high) cut-score on the screening test impacts the identification errors for the placement test.
The results show that setting the cut-score for the screener to the same selectivity as the identification test substantially reduces the number of “false positives” (students who meet the cutoff on the screener, but not the placement test). However, this comes at the expense of a greater number of false negatives, where students who would be successful on the placement test were passed over by the screener. As mentioned before, these false negative errors are more problematic because it means students are denied opportunities they could benefit from.
Implications for Practice
Researchers and practitioners have long been concerned with increasing the diversity and representation of gifted and talented programs. Research shows that referral-led identification processes may contribute to the problem of underrepresentation. Replacing a referral system with a universal screening policy is an important tool for addressing the underrepresentation of certain groups of students, particularly ethnic and racial minorities, low socioeconomic status (SES), and English learner students in gifted and talented programs. These results show some specifics of how that universal screening process might be implemented by school districts.
Our results and analyses are based on one school district’s results. When conducting screenings in your district, it’s best to make evidence-based decisions, whether using the research literature or your own past experience with the program, to tweak your identification system. Local data should be used to determine how liberal to be in setting cut-scores for the initial screening. This will ensure that students who would be successful are identified for further testing, while not expanding the testing pool too greatly.
Dr. Joni Lakin will be at the HMH booth at the NAGC annual convention to discuss CogAT in Charlotte, NC from November 9–12.