Into Math: Impact Study 2021–2022 School Year

This study included teachers from six public schools from two urban school districts. One school district is in the Northeast. Three schools from this district volunteered to participate in the study. The school district consists of approximately 19,000 students and 36 schools. Additionally, three schools from a Southeast urban school district also agreed to participate in the study. This school district serves over 52,000 students and consists of 52 schools.

During the 2021–2022 school year, all teachers were introduced to the HMH Connected Teaching and Learning System (CTLS). This integrated system incorporates a benchmarking assessment, Common Core-aligned core instruction, personalized supplemental practice, and meaningful professional learning all on one platform.

For this study, all teachers had access to four components of the CTLS:

1. The HMH Growth Measure Math for benchmarking
2. HMH Into Math^®, an innovative K–8 core math solution designed with an emphasis on productive perseverance in problem-solving and application of knowledge
3. The supplemental digital program Waggle^® that provides relevant practice to support Into Math core lessons and offers personalized and differentiated instruction at the right time for each student
4. Program-specific professional learning and coaching to ensure they had sufficient knowledge and skills to successfully implement the CTLS

Across all six schools, a total of 26 teachers agreed to participate in the study during the 2021–2022SY. Across these classrooms, there were a total of approximately 994 students. However, one school did not report demographic data.

This sample consisted of primarily female students (n= 475, 54.5%). The student population consisted of mostly African American students (47.0%) and White students (33.4%), with 14.5% Hispanic, 2% Asian, and remaining students reporting a different race/ethnicity.

Sixty-three percent of students were eligible for free or reduced-price lunch. Four percent of the sample was identified as English Language Learners, and 33.8% of students were labeled as Special Education students.

HMH GROWTH MEASURE

HMH Growth Measure is a computer-adaptive test (CAT) that adjusts item difficulty based upon students' responses. The assessment is designed to provide timely insights to support student grade-level placement decisions in fewer than 45 minutes per student, with the assessment consisting of approximately 35 items.

Student performance can be measured in terms of a scaled score. Scaled scores range from G1 to G99, where G represents the grade level of the test administered. The scaled score makes it possible to compare individual student performance and to aggregate student performance within grade level classrooms.

HMH Growth Measure provides benchmark assessments up to three times a year, and it is recommended that districts use HMH Growth Measure at the beginning of the year (Fall), in the middle of the year (Winter), and at the end of the year (Spring).

IMPLEMENTATION AND FIDELITY

Study teachers completed a bi-monthly implementation log to report on lessons completed, as well as the components and resources utilized for Into Math and Waggle. These logs were developed as a Google Worksheet to provide teachers with easy “anytime” access in a user-friendly format.

To confirm teacher participation in professional learning, attendance records maintained by HMH consultants and coaches were obtained and reviewed to determine which teachers participated in the various events scheduled over the course of the school year.

HMH INTO MATH AND WAGGLE

All teachers reported using the Into Math and Waggle programs for the duration of the 2021–2022 school year.

During this time, teachers reported spending an average of 102 minutes on math instruction. Teachers in the Northeast school district reported substantially more time in math instruction (M = 118 minutes) than teachers in the Southeast school district (M = 90 minutes).

Teachers completed an average of 73% or more of the Into Mathprogram modules and lessons during the school year.

While usage of Into Math was consistent across the sites, Waggle implementation varied within and across districts.

In the Southeast school district, most teachers reported using Wagglea few times per month. In the Northeast district, about a third of teachers reported using Waggle weekly, while the remaining teachers reported using the program a few times per month or few times per semester.

Teachers that indicated they used essential Into Math program components for most or all lessons on average were classified as having met minimum implementation expectations for Into Math. Further, teachers were classified as meeting Waggle implementation guidelines if they used the program for an average of 45 minutes per week.

PROFESSIONAL LEARNING

Teachers at both sites had access to a variety of professional learning opportunities focused on Into Math and Waggle, as well as best practices for general math instruction.

For the Northeast site, all teachers participated in at least two professional learning sessions (M = 3.2 sessions). Most teachers participated in a Getting Started training for Into Math and a Getting Started training for Waggle.

Further, a majority of teachers (73%) also completed one follow-up session, with 63.6% of teachers in the Northeast school district also receiving one additional coaching session.

While most teachers in the Northeast completed several professional learning sessions, researchers were only able to confirm participation in these activities for 20% of teachers from the Southeast district. These teachers all reported participating in at least one coaching session.

Given the inconsistent exposure to professional learning, researchers did not include attendance in these sessions when calculating fidelity to the Connected Teaching and Learning System model.

Thus, only usage of Into Math and Waggle were used to determine which teachers met implementation guidelines for the CTLS.

Of the 18 participating teachers for which complete implementation data was available, 11 teachers met the minimum implementation guidelines; 7 teachers did not meet the implementation guidelines.

Data was available for a total of 833 students who completed the beginning-of-year (BOY), middle-of-year (MOY), and end-of-year (EOY) HMH Math Growth Measure. Among these students, there was significant improvement at all three test administrations, F (2, 1664) = 173.0, p < .001.

Students made significant progress from BOY (M = 41.57 SD = 12.95) to EOY (M = 48.99, SD = 16.92) with an average increase of 7.4 scaled points.

Figure 1 displays the gains made on HMH Math Growth Measure Scaled Scores (SS) from the beginning of the year to the end of the year for each grade-level span. Follow-up analyses indicated that there was statistically significant growth (all ps < .05) at all grade-level bands.

Follow-up analyses also indicated that students of several subgroups, including students eligible for free/reduced-price lunch, students with Limited English Proficiency (LEP), and students receiving Special Education services all witnessed significant growth from the beginning of the year to the end of the year (all ps < .05).

Teachers participating in the study were classified as either meeting implementation guidelines for the CTLS (n = 11) or not meeting implementation guidelines (n = 7). Student BOY to EOY Math Growth Measure scores from classes that met implementation criteria were compared to those that did not meet the criteria.

As shown in Figure 2, there is a significant positive relationship between growth and implementation level on the Growth Measure, F (1, 627) = 275.03, p <.001.

Students in classrooms in which teachers met minimum implementation levels grew an average of about 10 points (BOY = 40.74 SS vs. EOY = 50.65 SS) over the course of the school year, while students in classes in which teachers did not meet the implementation criteria grew an average of about 6 points (BOY = 44.28 SS vs. EOY = 50.49 SS).

This exploratory impact study of the HMH Connected Teaching and Learning System for Math found that most teachers covered a majority of the program lessons and typically spent around 102 minutes a day engaged in math instruction.

While most teachers were able to implement the model with fidelity (61.1%), not all teachers were able to cover the required lessons and components in Into Math. Also, there were challenges when it came to teachers using Waggle, with usage varying greatly across participants.

Further, in one district, confirming professional learning attendance was also problematic.

Overall, teachers reported that use of the CTLS had a positive impact on students, with 70.8% of teachers indicating the system positively impacted critical thinking and problem-solving among students. A majority of teachers (56.5%) also reported the system had a positive impact on students’ productive perseverance.

Further, more than 75% of teachers indicated that Into Math contributed to marked improvement in students’ academic achievement.

Similarly, over 70% of teachers reported that Waggle helped students build understanding of content taught during core instruction. Teachers also found Waggle to be helpful for both struggling students (63.6%), as well as enriching for advanced students (77.3%).

While statistical analyses revealed that students using Into Math and Waggle did make significant progress from the beginning of the year to the end of the year, the level of growth was dependent on how teachers implemented the programs.

When examining the impact of the CTLS, analysis showed that students made significantly greater academic progress when teachers met the minimum implementation guidelines for Into Math and Waggleas compared to classrooms in which teachers did not implement the CTLS as prescribed.

This initial research provides encouraging results, suggesting that using the Connected Teaching and Learning System isassociated with improved student math achievement scores and improved teaching outcomes. More rigorous research is needed to determine the context and conditions in which the program's usage leads to greater increases in student achievement.