Math in Focus: Quasi-Experimental Examination of New York Math Test Scores

This study included 43 schools that purchased and adopted HMH Math in Focus^® 2020 during the 2019-2020 school year (Cohort A n= 29) and 2020-2021 school year (Cohort B n = 14). Researchers performed confirmation calls with schools/districts to determine the extent to which identified schools have used Math in Focus. These confirmation calls allowed researchers to determine that potential treatment schools were established Math in Focus users.

Given the nature of the study, researchers relied on self-report measures from confirmation calls to determine usage of Math in Focus 2020 as their core mathematics program in grades K-6. Data collected included (1) verification of the use of the Math in Focus 2020 program and at which grades and (2) the proportion of students within schools that used this curriculum.

Math in Focus^®: Singapore Math^® by Marshall Cavendish^® is an innovative mathematics program for grades K–8+accelerated that focuses on problem-solving and using model drawing to drive the acquisition and application of mathematical skills. The program leverages global best practices and research to create an approach based on visual models and engaging hands-on activities that empower students to develop critical thinking, positive attitudes, and the confidence needed to achieve in math. At all Math in Focus sites, a majority of students at each grade level were taught with the program as their core math instruction.

The study consisted of 120 schools, with 43 schools using HMH Math in Focus during the study period and 77 schools serving as the matched sample comparison group.

Given that performance data is available for each grade level, the unit of analysis is grade-level performance within schools. Therefore, Math in Focus schools were matched by school-level characteristics and grade-level performance measures to unique schools within the state.

Further, schools in Cohort A had been exposed to prior versions of Math in Focus, with some schools using the program dating back to the 2012-2013 school year. Schools in Cohort B were completely new to Math in Focus in 2020-21 and had not used any older copyright of the program.

Table 1 below displays the final grade-level units for treatment and comparison schools included in the final analytic sample.

New York State Tests: The New York State Testing Program (NYSTP) has been used since Spring 2013 to measure student knowledge and skills as defined by grade-level New York State Learning Standards in ELA and Mathematics for grades 3-8. The tests are criterion-referenced, composed of multiple-choice (MC) and constructed-response (CR) test items based on the New York State P–12 Learning Standards. Session 1 consists of multiple-choice questions, while Session 2 consists of multiple-choice, short-response, and extended-response questions.

The performance-level cut scores were initially established in 2013 but were revised in 2018 when the test design changed, and the test length was reduced. Due to the variability in which instruction was provided to students in the 2020-21 school year in response to COVID-19, the 2021 Grades 3–8 ELA and Mathematics tests were shortened further only to include multiple-choice items (Session 1).

The present study uses math assessment data from the NYSTP. According to the NY Department of Education, the tests assess students’ conceptual understanding, procedural fluency, and problem-solving abilities rather than evaluating their knowledge of isolated skills and facts.

The tests were designed to allow the classification of student proficiency into four performance levels (Level I, Level II, Level III, and Level IV). For the present study, percent proficient and above (i.e., Levels III and IV) is the student achievement measure.

Propensity Match Procedures: To evaluate the impact of Math in Focus 2020, a comparison group was identified using Propensity Score Matching (PSM). PSM was conducted separately for each grade level, within a school, and a cohort.

To create these comparison groups, researchers conducted a three-step process:

Step 1. Computing Propensity Scores: For each unit included in a model, the probability of receiving the Math in Focus treatment was estimated using a logistic regression model with the Math in Focus treatment flag as the outcome variable and grade-level characteristics as predictor variables.

Step 2. Matching: PSM was implemented with MatchIt version 4.5.0 (Ho et al., 2011) using the nearest-neighbor matching method, with a generalized linear model specified as the distance measure and logit specified as the link function. Each PSM model’s Math in Focus unit was sequentially matched with one potential control unit.

Step 3. Assessing Balance: Baseline equivalence was assessed on key characteristics using a two-sided Welch’s Two Sample t-test with a 0.05 alpha level.

Statistical analyses were performed to evaluate the match between Math in Focus students and comparison-group students on 11 characteristics (e.g., number of students, percent English language learners).

Overall, these analyses revealed two significant (p < .05) differences between the groups. The combined Math in Focus 2020 sample had a significantly higher Baseline Proficient rate on the 2019 NYSTP math assessment (M = 73%) when compared to the comparison group (M = 63%). In addition, the combined Math in Focus 2020 sample consisted of a larger proportion of Hispanic students (M = 16.0%) than the comparison group (M = 8.15%). Thus, researchers controlled for these two characteristics in all analyses.

Correlational analyses were performed to examine if there was a significant relationship between years of Math in Focus usage and student change in math performance, measured by the difference between 2022 and 2021 proficiency rates.

Analyses were performed across both cohorts so that years of usage included schools with 1 to 8 years of usage. Further, given shifts in performance trends, especially before and after the pandemic, the examination was focused on the change from Spring 2021 to Spring 2022.

Results showed a significant moderate relationship (r (41) = 0.36, p < .05) such that more experienced Math in Focus schools (from 3+ years of usage) demonstrated more positive change in math achievement from Spring 2021 to 2022.

As previously stated, given the nature of program purchases and the state testing schedule, researchers were able to identify two different cohorts of Math in Focus 2020 users exposed to the program for various lengths of time.

Given the initial finding that the length of exposure to prior versions of Math in Focus was associated with gains in NYSTP math scores, the analysis plan consisted of comparing different cohorts to determine if the amount of time students were exposed to the new version of the program had an effect.

There were statistical trends in which cohorts using Math in Focus 2020 had higher proficiency rates after one year and two years of usage. However, this finding was only statistically significant (p < .05) after three years of use when assessing Spring 2022 proficiency rates.

An Analysis of Covariance (ANCOVA) examining the differential impact of Math in Focus 2020 on student math scores revealed that schools in Cohort A, using Math in Focus 2020 for three years, had a significantly higher percentage of students at or above proficiency (M = 55.2% adjusted) than comparison schools (M = 51.6% adjusted), F(1,289) = 5.82, p < .05.

Post hoc tests revealed that this finding was significantly different at grade 4 (p < .05), with positive trends at grade 3 and grade 5.

Subgroup analyses were also performed to determine if Math in Focus had a differential impact on specific student populations. Across all groups, Math in Focus students with three years of program usage outperformed similar students with statistically different results for White students, Hispanic students, and students with disabilities (see Table 3).

Results from the Math in Focus 2020 quasi-experimental study showed that Math in Focus usage is associated with positive effects on student math performance in New York.

Math in Focus 2020 schools demonstrated higher proficiency rates than control schools across all years examined; statistically significant differences were observed in Post Year 3 (p < .05). Additionally, results by years of usage showed that Math in Focus schools with greater experience demonstrated more positive changes from 2021 to 2022.

Examination of results within each grade level also showed a similar positive pattern of results. Math in Focus users generally demonstrated higher proficiency rates within all grade levels. Across all subgroups analyzed, including males, females, Whites, Hispanics, African Americans, students with disabilities, economically disadvantaged students, and English language learners, Math in Focus students demonstrated higher proficiency rates on average than non-users.

While the results are encouraging, the study has several limitations that should be considered when interpreting results. The NY state education departments mask data for students in subgroups with less than ten students; the data from Math in Focus 2020 schools do not represent all impacted students.

Further, the study uses assessment data from before and after the COVID-19 pandemic. As such, results from this study should be considered in conjunction with other education impact analyses that took place around this time to understand better how COVID-19 might have influenced standardized assessment scores.

Researchers were unable to obtain implementation data on how well teachers implemented the program and their level of fidelity to the learning model. Such data could reveal more substantial effects, as program implementation is a known mediator when conducting curricula research.

Lastly, quasi-experimental studies like these rely on existing users. Thus, the analysis must partially rule out other potential external factors leading to these results. However, the statistical procedure put into place, and the consistency of the findings, do increase the confidence that the usage of Math in Focus was the underlying mechanism leading to the significant differences between treatment and comparison.

Despite these limitations, the results from this quasi-experimental study using state assessment data provide evidence that Math in Focus 2020 is an effective math program that can help accelerate student learning.