Simi Valley Unified School District located in California enrolls approximately 18,000 students in Grades K through 12. The demographic breakdown of students in Simi Valley are 46% White, 37% Hispanic, 10% Asian/Pacific Islander, 4% two or more races, less than 1% Black, and less than 1% Native American. Thirty-four percent of students are eligible for free or reduced-price lunch, and 11% are English Language Learners.
During the 2016–2017 school year, students from four middle schools in Simi Valley were selected to participate in a study led by a third-party firm, RMC Research. The evaluation sought to answer the following three research questions:
- To what extent did students’ mathematics assessment scores improve after participating in Math 180?
- How did dosage influence Math 180 students’ assessment score gains?
- How did Math 180 differentially affect demographic subgroups in terms of students' assessment score gains?
Math 180 is a program designed to assist middle school students who are struggling with multiplication, division, fractions, and decimal operations. This dedicated intervention class is in addition to the students' core mathematics instruction. Math 180 was implemented in a daily 45- to 55-minute block: whole class instruction occurs during the first 5 minutes and is followed by two 20- to 25-minute rotations. During one rotation, students receive adaptive instruction through the student application and practice, and during the other rotation, students participate in teacher-led small-group instruction that incorporates technology, differentiated instruction through Boost and Stretch lessons and the use of High-Leverage Practices, and Mindset activities. Student progress is monitored and assessed through software elements, mSkills tests that measure mastery of key concepts, and administration of the HMH Math Inventory® three times per year (fall, winter, spring). A Mindset Scan measures students’ attitudes toward mathematics at the beginning and end of the program.
Simi Valley Unified School District serves approximately 18,000 students in kindergarten through Grade 12 and has 18 elementary schools, three middle schools, and four high schools. This study included 63 students in Grade 7 and 47 students in Grade 8 who were enrolled in Math 180 Course I or Course II. These students were chosen from three middle schools during the 2016–2017 school year. Table 1 presents the demographic characteristics of the student sample.
MATH INVENTORY
The Math Inventory is a computer adaptive assessment that measures students’ readiness for mathematics instruction using the Quantile® Framework for Mathematics, which utilizes the same scale for the difficulty of mathematical topics and concepts and a student’s knowledge of the prerequisite mathematics needed to be successful when learning these topics and concepts. The five strands of the Quantile Framework are based on the five content standards of the National Council of Teachers of Mathematics (NCTM) standards (i.e., number and operations, geometry, algebra/patterns and functions, data analysis and probability, measurement). The district administered the Math Inventory in spring 2016 and spring 2017 to Math 180 students.
SMARTER BALANCED SUMMATIVE ASSESSMENT
The Smarter Balanced summative assessment is a comprehensive end-of-year test that is aligned with the Common Core State Standards (CCSS) and measures progress toward college and career readiness. The test capitalizes on the strengths of computer adaptive testing—efficient and precise measurement across the full range of achievement and the timely turnaround of results.
In this study, students’ mathematics skills were measured by the Smarter Balanced mathematics summative assessment. The district administered the computer adaptive test in spring 2016 and spring 2017 to Math 180 students. Results are reported in scale scores, which fall on a continuous scale (from approximately 2,000 to 3,000) and increase across grade levels. Table 2 presents the Smarter Balanced mathematics summative assessment scale score norms for Grades 6–8.
RMC Research conducted a paired sample t-test to evaluate the extent to which students’ scores on the Math Inventory improved after participating in Math 180. Table 3 displays Math 180 students’ 2016–2017 Math Inventory fall and spring Quantile measures by grade level. On average, students in both Grade 7 and Grade 8 made significant Math Inventory gains between fall and spring.
In order to understand the factors that may be influencing growth, the study team examined the relationship between program usage and growth on the Math Inventory, controlling for baseline Math Inventory score and demographic characteristics. Regressions were conducted separately for Grade 7 and Grade 8 students and examined the contribution of Math 180 dosage (i.e., number of Math 180 topics completed) on Math Inventory progress.
For both Course I and Course II, students in Grade 7 and Grade 8, respectively, demonstrated greater Math Inventory progress with greater number of Math 180 topics completed. For every additional topic completed, Grade 7 students scored 12.95 points higher on the spring 2017 Math Inventory, and Grade 8 students scored 10.45 points higher.
RMC Research conducted a paired sample t-test to evaluate the extent to which students’ Smarter Balanced assessment scores improved after participating in Math 180. Table 4 presents Math 180 students’ spring 2016 and spring 2017 scale scores and scale score change. Overall, Grade 7 students exhibited a significant scale score increase of 26.40 between spring 2016 and spring 2017. Grade 8 students exhibited no change during that same time period.
Regression analyses were used to examine the relationship between Math 180 dosage (i.e., number of Math 180 topics completed) on Smarter Balanced mathematics summative assessment progress, controlling for baseline score and demographic characteristics.
Results of the analyses for Course I showed that the number of Math 180 topics completed significantly contributed to gains in Grade 7 students’ Smarter Balanced scores from spring 2016 to spring 2017. Similarly, for Course II, results showed that number of Math 180 topics completed significantly contributed to gains in Grade 8 students’ Smarter Balanced scores from spring 2016 to spring 2017. For every additional topic completed, Grade 7 students scored 3.18 points higher on the spring 2017 Smarter Balanced assessment, and Grade 8 students scored 4.22 points higher.
The finding that dosage contributed to progress on both the Math Inventory and Smarter Balanced mathematics summative assessment led to a subsequent analysis to examine whether the relationship between dosage and progress was constant across all dosage levels or whether a minimum dosage was necessary to see significant progress.
To assess this, RMC Research established a low- and high-dosage group using completion of 10 topics as a cutoff. In each grade, the high-dosage group included students who completed more than 10 topics, and the low-dosage group included students who completed 10 or fewer topics. Analysis compared change in assessment scores across low- and high-dosage groups. Students’ baseline scores were statistically equivalent between low- and high-dosage groups for both grade levels for both assessments.
MATH INVENTORY GROWTH
Table 5 presents Math 180 students’ fall 2016 and spring 2017 Math Inventory scores and change in scores for each of the low- and high-dosage groups. Students in all groups made significant fall-to-spring gains on the Math Inventory. Results showed that Grade 7 students who completed more than 10 topics made significantly greater Math Inventory score gains than students who completed 10 or fewer topics (Ms = 215.00 and 109.00, respectively). Similarly, Grade 8 students who completed more than 10 topics made significantly greater Math Inventory gains than students who completed 10 or fewer topics (Ms = 240.00 and 152.14, respectively).
SMARTER BALANCED ASSESSMENT GROWTH
Table 6 presents Math 180 students’ spring 2016 and spring 2017 Smarter Balanced scale scores and scale score change for each of the low- and high-dosage groups. Results showed that students in both Grade 7 and Grade 8 who completed more than 10 topics made significantly greater Smarter Balanced scale score gains than students who completed 10 or fewer (Grade 7 Ms = 35.82 and 10.40, respectively, and Grade 8 Ms = 23.89 and -42.13, respectively).
To address the question of whether Math 180 differentially affects students of different subgroups in terms of their growth, regression analyses were used to examine the relationship between student demographic characteristics on Math Inventory and Smarter Balanced mathematics summative assessment progress. Regressions were conducted separately for Grade 7 and Grade 8 students, controlling for baseline score, eligibility for free and reduced-price lunch, and identification as non-White. Findings showed no significant differences in Grade 7 or Grade 8 students’ Math Inventory or Smarter Balanced score gains across demographic subgroups.
This study of 63 Grade 7 and 47 Grade 8 students who were enrolled in Math 180 in 2016–2017 sought to measure how Math 180 students performed on the Math Inventory and Smarter Balanced mathematics summative assessment after participating in Math 180. Findings revealed that the overall sample of Grade 7 Math 180 students made significant progress from spring 2016 to spring 2017 on the Smarter Balanced mathematics summative assessment; however, Grade 8 students did not. Students in both grades made significant gains on the Math Inventory between fall 2016 and spring 2017. Assessment score change on both assessments was similar across demographic subgroups.
Analyses of assessment score change in the context of Math 180 dosage—defined by number of topics completed—revealed that students who completed more topics made greater gains on both the Smarter Balanced mathematics summative assessment and the Math Inventory. Further analyses revealed that students who completed a minimum number of topics—the high-dosage group—had assessment change patterns that differed from students in the low-dosage group. Specifically, in both grades students who completed more than 10 topics demonstrated significant gains in Smarter Balanced summative assessment scores, whereas students in the low-dosage group did not demonstrate gains. On the Math Inventory, students in both the low- and high-dosage groups made gains in each grade; however, in both Grade 7 and Grade 8, students in the high-dosage group made significantly greater gains than students in the low-dosage group. These findings suggest that students who were able to fully engage in Math 180 were successfully able to achieve progress in mathematics knowledge and skills as measured by both a standardized mathematics assessment and HMH’s Math Inventory. These findings also have implications for programmatic application—to start to see effects, students may need to complete a minimum number of topics. RMC Research recommends further investigation of the concept of minimum Math 180 dosage using additional HMH data sets.