System 44: Charles Armstrong School 2018–2019
At a glance
  • Promising Evidence
  • Program: System 44®
  • Subjects: Intervention Curriculum, Literacy
  • Report Type: Efficacy Study, Study Conducted by Third Party
  • Grade Level: Elementary, Middle
  • Region: West
  • Population: Students with Disabilities, Students with Specific Learning Disabilities, Students with Dyslexia
  • District Urbanicity: Suburban
  • District Size: Small
  • District:: Charles Armstrong School, CA
  • Participants: 45
  • Outcome Measure: System 44 Software Use, Phonics Inventory, Reading Inventory, Word Identification Spelling Test, Read Naturally Reading Fluency, Track My Progress Reading Comprehension
  • Implementation: Software-only (20 minutes two to five times per week)
  • Evaluation Period: 2018–2019
  • Study Conducted By: Forge Research Group
The Challenge

Students who do not meet or exceed benchmark scores on state and national tests are less likely to graduate from high school, are less likely to persist in or successfully complete future academic and workplace training endeavors, and are overall less likely to be on track for future academic and workplace success (CCRSC, 2013). Compared to other students, students with language-based learning differences such as dyslexia are 2 ½ times more likely to fail grade-level benchmark tests by 4th grade and three times more likely to fail tests by 8th grade (National Center for Education Statistics, 2013). The Charles Armstrong School identified the need to provide an additional reading intervention program to accelerate Grades 3 to 7 students who have been diagnosed with dyslexia to grade-level reading proficiency.

The Solution

System 44 is a literacy intervention program designed to build foundational reading and decoding skills for the most challenged readers. This blended learning solution includes explicit, systematic instruction in foundational literacy skills through whole-group instruction, small-group targeted differentiated instruction, personalized learning through the software, and an independent reading library. Materials feature modeled and independent reading with age-appropriate books that target decoding skills and strategies to promote comprehension, build vocabulary, and increase content-area knowledge. Using adaptive technology, students complete a personalized learning progression through five strands that include The Code, Word Strategies, Sight Words, Success, and Writing.

The Study
School Characteristics

The Charles Armstrong School in Belmont, California, is a private school designed to educate students with language-based learning differences such as dyslexia. The Charles Armstrong school serves approximately 240 second through eighth grade students1. It serves students with a range of ethnic backgrounds: African American (2%), Asian (2%), Caucasian (78%), Hispanic (4%), Native American (2%), and students with multiple ethnic backgrounds (12%). In the 2018–2019 school year, 25% of students received financial aid.

Participants

All students who completed System 44 instruction and a pretest and posttest (n=45) were included in the analysis of Phonics Inventory scores. An additional 18 students did not complete a Phonics Inventory posttest; these students are included in analyses involving school-collected reading metrics. Student demographics for the entire sample (N=63), which do not vary markedly from the demographics of the smaller sample used for Phonics Inventory analyses, are below (see Table 1; see Appendix Table 1 for the demographics of the smaller sample).

Implementation

Charles Armstrong School students in Grades 3 to 7 who were identified as struggling with foundational reading skills completed the Phonics Inventory and placed in the System 44 intervention. Students completed approximately 20 minutes in the personalized online student application between two and five days a week. Students continued to receive standard core reading instruction in the classroom (utilizing the Wilson curriculum and program materials) in addition to System 44 instruction.

Teachers met with two HMH consultants for an overview of the System 44 program and ongoing instructional coaching sessions while implementing System 44 during the 2018-2019 school year.

Measures
Software Usage Data

Student software usage data was collected as students used the online student application during System 44 instruction. Software usage data included number of completed series, number of completed topics, number of completed sessions, average time spent in each session, and number of sessions averaged per week.

Phonics Inventory

The HMH Phonics Inventory measures proficiency in the foundational reading skills of phonological decoding and sight word reading for students in Grades 3 to 12. The Phonics Inventory is used to identify whether students with low reading comprehension achievement also lack the skills needed to decode new words (leading to placement in System 44) or are best served by an intervention to develop reading comprehension strategies, text analysis skills, and background knowledge (leading to placement in other interventions). Assessment results include an accuracy score (range of 0–60) based on accurately reading sight words and nonsense words, a fluency score (range of 0–60) based on reading accurately as well as quickly, and a decoder status (pre-decoder, beginning, developing, or advancing). Phonics Inventory subscores are provided for Letter Recognition (range of 0–11), Nonsense Word Accuracy (range of 0–30), Sight Word Accuracy (range of 0–30), Sight Word Fluency (range of 0–30), and Nonsense Word Fluency (range of 0–30). Charles Armstrong students completed the Phonics Inventory before beginning instruction in System 44, and again at the end of the school year.

Read Naturally Reading Fluency

The Read Naturally Reading Fluency Progress Monitor is an efficient, valid, and reliable assessment to measure a student's progress when reading text aloud. The teacher listens to a student read a leveled passage at his/her instructional level (Grades 1–8) for one minute and determines the student's words correct per minute. Scores are graphed throughout the year to track student progress and make educational decisions.​ Charles Armstrong School students were assessed for Oral Reading Fluency using the Reading Fluency Progress Monitor multiple times during the 2018–2019 school year.

Track My Progress Reading Comprehension

Track My Progress is an online, computer-adaptive test designed to assess math and reading skills aligned to the Common Core State Standards in kindergarten through eighth grades. Track My Progress tests have been shown to be both reliable and valid measures of Common Core State Standard proficiency, normed to a nationally representative US sample. Student progress in meeting skills is tracked as students take four 20-minute tests each year, and teachers have access to reports on specific subject and domain proficiency. Charles Armstrong School student data included a spring 2019 percentile score (range 0–99) for Reading Comprehension. A percentile score above 41 is considered to reflect grade-level ability.

Word Identification Spelling Test

The Word Identification Spelling Test is a nationally standardized, individually administered diagnostic test that assesses students' fundamental literacy skills. Three subtests (Word Identification, Spelling, and Sound-Symbol Knowledge) and a composite score called the Fundamental Literacy Index are reported. The Word Identification subtest measures word reading accuracy which includes (a) students' sight recognition of familiar words and their ability to apply word attack skills in order to decode unfamiliar words and (b) their sight recognition or orthographic memory of high-frequency words with one or more irregularities. The Spelling subtest assesses students' ability to spell words correctly from dictation and specifically measures students' (a) recall of correct letter sequences for familiar words or one's ability to apply sound/symbol relationships and rules of English orthography in order to spell unfamiliar words and (b) their recall of letter order in high-frequency words with one or more irregularities. The Sound-Symbol Knowledge subtest assesses a student's ability to associate sound(s) (i.e., phonemes) with specific letter(s) (i.e., graphemes). The Word Identification Spelling Test is both valid and reliable for its stated purposes. Charles Armstrong School student data included a spring 2019 percentile score (range 0–99) for Word Identification, Spelling, Sound-Symbol Knowledge, and the Fundamental Literacy Index.

Reading Inventory

The HMH Reading Inventory measures reading comprehension proficiency for students in Grades K to 12. The Reading Inventory uses adaptive technology to determine a student’s reading comprehension level on the Lexile Framework for Reading; the higher the Lexile score, the more challenging reading material the student can comprehend. Test item difficulty ranges from items appropriate for developing readers to items requiring a reading proficiency indicating preparedness for college level texts, allowing measurement of skill growth regardless of the students’ initial ability. Assessment results include a Lexile scale score that indicates reading ability at a level of text complexity and a performance level of below basic, basic, proficient, or advanced, indicating achieved reading comprehension compared to grade-level expectations. Charles Armstrong School students completed the Reading Inventory in the spring of 2019, following System 44 instruction.

Results
Usage

Charles Armstrong School teachers began implementation of System 44 at different time points throughout the 2018–2019 school year (see Figure 1), and as such, length of student participation varied greatly. Third grade students began in February 2019, 4th grade students began in March 2019, 5th grade students began in April 2019, 6th grade students began in November and December 2018 and others started in March and May 2019, and 7th grade students began in November 2018 and some in May 2019.

Wf1315368  Figure 1
FIGURE 1. Charles Armstrong School System 44 Students, Grades 3–7, (N=63) Number of Students Beginning System 44 by Month, 2018–2019
Wf1315368  Figure 1

Students who participated in System 44 during the 2018–2019 school year (see Table 2) completed an overall average of 2.9 System 44 series (SD=2.7) and 34.2 topics (SD=24.4) over an average of 28 total sessions (SD=18.8), averaging 2.1 (SD=0.6) sessions per week and totaling 521.7 minutes (SD=408.3) in the System 44 online student application.

Performance

An independent evaluator from Forge Research Group analyzed student academic achievement using test score data provided by the Charles Armstrong School and program usage data provided by Houghton Mifflin Harcourt. System 44 students’ English Language Arts (ELA) performance was examined pre- and post-implementation using multiple independent measures of reading. This analysis included scores on the Phonics Inventory and the Read Naturally Reading Fluency Progress Monitor. System 44 students demonstrated significant gains in ELA proficiency during the 2018–2019 school year on each of these reading measures. Notably, because of the self-paced nature of the program and the variability in participation, individual student gains were inconsistent.

System 44 students who completed the Phonics Inventory multiple times demonstrated statistically significant overall gains in Phonics Inventory fluency scores. Students increased from an average of 8.8 on first assessment to 11.2 on the second assessment (see Figure 2). When disaggregated by grade level, 3rd and 4th grade students demonstrated a significant overall gain in Phonics Inventory fluency scores. Fifth grade students completed the least amount of time in System 44 (M=121 minutes compared to the overall M=522 minutes) and had 2 outlier students with large pretest to posttest decreases, resulting in non-significant overall decreases in Phonics Inventoryfluency scores (mean change was -0.40 points when outliers were removed, indicating that participation was not long enough to increase reading fluency). Sixth and 7th grade students demonstrated notable gains in Phonics Inventory fluency scores, however, inconsistent time spent using System 44 resulted in inconsistent student gains, decreasing the overall statistical significance. Three students’ grade level was not identified and so are only included in the overall results.

Wf1315368  Figure 2
FIGURE 2. Charles Armstrong School System 44 Students, Grades 3–7, (n=45) Change in Phonics Inventory Fluency Score, Overall and by Grade, 2018–2019

Note. *statistically significant change at p<.05. The increase in average Phonics Inventory Fluency score was statistically significant for System 44 students overall (t=2.48, p=0.02), and for third (t=2.93, p=0.01) and fourth grade students (t=2.70, p=0.03).

Wf1315368  Figure 2

In addition to Phonics Inventory fluency score gains, System 44 students also achieved increased pretest to posttest decoder status on the Phonics Inventory (see Figure 3). The percentage of students achieving an “Advancing” decoder status on the Phonics Inventory, indicating readiness to move to a more complex reading intervention, increased overall from 4.4% to 11.1%, while the percentage of students scoring at a “Pre-Decoder” decoder status, indicating little or no foundational reading skills, decreased from 2.2% to 0% during the 2018–2019 school year.

Wf1315368  Figure 3
FIGURE 3. Charles Armstrong School System 44 Students, Grades 3–7, (n=45) Change in Phonics Inventory Decoder Status, 2018–2019
Wf1315368  Figure 3

Of note, approximately 25% of the System 44 students who began the program with a Developing decoder status progressed to an Advancing decoder status, indicating readiness to graduate from the System 44 intervention. Large differences in mean topics completed and time spent in the software (see Table 3) distinguished between students who progressed from Developing to Advancing (mean topics completed=74.5, mean software time=995.5 minutes) and students who remained as Developing (mean topics completed=42.2, mean software time=526.6 minutes).

Phonics Inventory subscores also demonstrated significant literacy gains during the 2018–2019 school year (see Figure 4). At pretest, Charles Armstrong School students demonstrated proficiency in Letter Name Accuracy (mean score 10.87 out of 11), an indication that students had the prerequisite skills to build decoding skills. From pretest to posttest, System 44 students achieved statistically significant gains in both Sight Word Accuracy (2.33 points), and Nonsense Word Accuracy (2.40 points), indicating an increase in decoding skills. Students also achieved statistically significant gains in both Sight Word Fluency (1.18 points) and Nonsense Word Fluency (1.28 points), indicating progress in foundational reading skills. Importantly, the gains in accuracy and fluency for both sight words and nonsense words indicate that System 44 is an effective intervention for students with dyslexia.

Students with dyslexia often fit one of two identified scoring trends2, having one of the following reading profiles: 1) possessing sight word reading skills but not nonsense word reading skills, indicating compensated memorization of words and a lack of true decoding skills; or 2) possessing neither sight word nor nonsense word reading skills, indicating both a lack of decoding skills and little experience with reading the irregular words and vocabulary specific to the English language. According to the pretest subscores, 7 students demonstrated the first profile and 18 students demonstrated the second profile. By posttest, 6 of the 7 profile 1 students increased in nonsense word accuracy and 17 of the 18 profile 2 students increased both sight word and nonsense word accuracy, indicating that all students increased decoding skills rather than continuing to compensate with memorization.

Wf1315368  Figure 4
FIGURE 4. Charles Armstrong School System 44 Students, Grades 3–7, (n=45) Change in Phonics Inventory Sub Scores, 2018–2019

Note. *statistically significant change at p<.05. The increase in average Phonics Inventory subscore was statistically significant for Sight Words Accuracy, (t=4.32, p=0.00), Sight Words Fluency (t=2.36, p=0.02), Nonsense Words Accuracy (t=3.26, p=0.00), Nonsense Words Fluency (t=2.05, p=0.05), and Total Accuracy (t=5.51, p=0.00).

Wf1315368  Figure 4

Further, increased use of the System 44 online software was a statistically significant predictor of Phonics Inventory Fluency score growth, both before and after correcting for selection bias (see Appendix Table 3 for details). On average, System 44 students who completed more topics in the System 44 online software3 also achieved higher gains in the Phonics Inventory Fluency score after accounting for initial Fluency scores (see Figure 5).4

Wf1315368  Figure 5
FIGURE 5. Charles Armstrong School System 44 Students, Grades 3–7, (n=43) Phonics Inventory Fluency Score by Implementation, Spring 2019

Note. Minimal=15 or fewer topics; Moderate=16 to 29 topics; High=30 or more System 44 topics completed. Number of System 44 topics completed was a significant predictor of Phonics Inventory Fluency Score gain after accounting for initial Fluency Score (Standardized B=0.41, F=5.72, p=0.01, Adjusted R2=0.24).

Wf1315368  Figure 5

Notably, disaggregation of the data indicated that System 44 was associated with significant Phonics Inventory Fluency Score gains for most categories of students during the 2018–2019 school year (see Figure 6). When results were disaggregated by gender, both males and females achieved statistically significant Phonics Inventory gains from pre- to post-instruction. When results were disaggregated by ethnicity, Caucasian students achieved statistically significant Phonics Inventory gains. Although the sample was too small (n=6) to capture the true significance of gains statistically, multiracial students gained an average of 4.7 points on the Phonics Inventory, in line with other students’ gains. Other categories of students had fewer than 4 members and so results were suppressed.

Wf1315368  Figure 6
FIGURE 6. Charles Armstrong School System 44 Students, Grades 3–7, (n=45) Change in Phonics Inventory Fluency Score, Overall and by Student Category, 2018–2019

Note. *statistically significant change at p<.05. The increase in average Phonics Inventory Fluency Score was statistically significant overall, (t=2.48, p=0.02), for male (t=2.47, p=0.02) and female students (t=2.31, p=0.03), and Caucasian students (t=2.15, p=0.05). Categories with fewer than 4 students were suppressed.

Wf1315368  Figure 6

In addition to increases on the Phonics Inventory, System 44 students demonstrated statistically significant growth in Oral Reading Fluency (ORF) on the Read Naturally Reading Fluency Progress Monitor. From the beginning to end of the 2018–2019 school year, System 44 students averaged a 60% increase in words correct per minute (wcpm) on the ORF test (see Figure 7). When disaggregated by grade level, 3rd, 4th, and 5th grade students all demonstrated significant gains in the ORF, gaining 34, 29, and 30 wcpm, respectively.

Wf1315368  Figure 7
FIGURE 7. Charles Armstrong School System 44 Students, Grades 3–5, (n=35) Change in Oral Reading Fluency Score, Overall and by Grade, 2018–2019

Note. *statistically significant change at p<.05. The increase in average Oral Reading Fluency score was statistically significant for System 44 students overall (t=14.13, p=0.00), and for third grade (t=8.34, p=0.00), fourth grade (t=10.59, p=0.00), and fifth grade students (t=5.75, p=0.00).

Wf1315368  Figure 7

System 44 students completed the Reading Inventory at the end of the school year in May 2019. The Reading Inventory is sensitive enough to capture below-grade-level ELA ability and growth not usually captured by other assessments (including negative Lexile scores). System 44 students who completed the Reading Inventory in the spring averaged a 410 Lexile (L) Score (see Figure 8), with extreme variation between students in different grades (from 180L to 784L).

Wf1315368  Figure 8
FIGURE 8. Charles Armstrong School System 44 Students, Grades 4–7, (N=36) Average Reading Inventory Score Overall and by Grade, Spring 2019
Wf1315368  Figure 8

Charles Armstrong School System 44 students also completed multiple reading metrics at the end of the school year. In addition to the HMH Phonics Inventory and Reading Inventory, students completed the Word Identification Spelling Test (WIST), the Track My Progress Reading Comprehension assessment, and the Read Naturally ORF assessment. Spring 2019 scores on each of the valid, nationally normed school reading metrics demonstrated a statistically significant correlation with each of the HMH reading metrics. While pretest and posttest scores were not available to directly test the relationship between gains on HMH reading metrics and the WIST or Track My Progress assessments for the 2018–2019 school year, this indicates that System 44 students’ increases in HMH reading metrics will likely be correlated with increases in school reading metrics in subsequent years.

Wf1315368 Table 4
Note. * indicates p<.05; ** indicates p<.01; WIST=Word Identification Spelling Test; wcpm=words correct per minute.
Wf1315368 Table 4
Conclusion

Multiple independent measures support the idea that students who received System 44 instruction made statistically significant improvements in English Language Arts and Literacy achievement. After one year of instruction, students in Grades 3 to 7 at the Charles Armstrong School demonstrated statistically significant increases in Phonics Inventory fluency scores. Students also increased in assessment performance levels; the percentage of students achieving an Advancing decoder status on the Phonics Inventoryincreased overall from 4% to 11%. Students achieved statistically significant increases in Phonics Inventory subscores measuring both sight word and nonsense word fluency, indicating growth in decoding skills rather than compensated memorization, as is often seen in students with dyslexia. Disaggregation of the data by gender indicated that use of System 44 was associated with significant gains in Phonics Inventory fluency scores for both male and female students. Students worked through varying amounts of the self-paced, personalized online instruction components – increased completion of System 44 topics was a significant predictor of Phonics Inventory fluency score gains during the 2018–2019 school year. In addition to gains measured by HMH metrics, System 44 students also demonstrated statistically significant gains in Oral Reading Fluency from beginning to the end of school year as measured by the Read Naturally Reading Fluency Progress Monitor. This study demonstrates that using System 44 to provide a reading intervention is an effective method of increasing literacy for students struggling to achieve grade-level ELA proficiency.

  1. Charles Armstrong School Quick Facts found at www.CharlesArmstrong.org.
  2. Scholastic Phonics Inventory Data Interpretation Guide accessed at http://52.1.239.6/product-supp....
  3. Graph excludes 2 multivariate outliers (students who completed a large number of topics and showed large score decreases).
  4. Of note, the Charles Armstrong School System 44 student data showed a strong correlation between pretest score and number of topics completed, with students with the lowest pretest scores also completing the fewest topics. This pattern was found because 3rd and 4th grade students had lower pretest scores than 6th and 7th grade students (as would be expected), and the minimal and moderate implementation groups primarily included 3rd and 4th grade students while the high implementation group primarily included 6th and 7th grade students. Further confounding the relationship, students in lower grades and with lower pretest scores generally make more gains that students in higher grades and with higher pretest scores. As such, the relationship between fluency gains and number of topics completed (M=3 for each implementation group) is more similar across implementation group than would be expected in a population in which students in all grades participated in System 44 for equal amounts of time.
References

College and Career Readiness and Success Center (CCRSC). (2013). Predictors of Postsecondary Success. Washington, DC: American Institute for Research.

National Center for Education Statistics. (2013). The Nation’s Report Card: A First Look: 2013 Mathematics and Reading (NCES 2014-451). Institute of Education Sciences, U.S. Department of Education, Washington, D.C.

Appendix
Wf1315368 Table A2
Note. M=Mean; SD=Standard Deviation; n=sample size; 95% CI=95% Confidence Interval; df=degrees of freedom; p=significance.
Wf1315368 Table A2
Wf1315368 Table A3
Note. N=sample size; M=Mean; SD=Standard Deviation; SB=Standardized Beta; 95% CI=95% Confidence Interval; p=significance.*Final model also includes these covariates: gender.
Wf1315368 Table A3

1. Charles Armstrong School Quick Facts found at www.CharlesArmstrong.org.

2. Scholastic Phonics Inventory Data Interpretation Guide accessed at http://52.1.239.6/product-supp....

3. Graph excludes 2 multivariate outliers (students who completed a large number of topics and showed large score decreases).

4. Of note, the Charles Armstrong School System 44 student data showed a strong correlation between pretest score and number of topics completed, with students with the lowest pretest scores also completing the fewest topics. This pattern was found because 3rd and 4th grade students had lower pretest scores than 6th and 7th grade students (as would be expected), and the minimal and moderate implementation groups primarily included 3rd and 4th grade students while the high implementation group primarily included 6th and 7th grade students. Further confounding the relationship, students in lower grades and with lower pretest scores generally make more gains that students in higher grades and with higher pretest scores. As such, the relationship between fluency gains and number of topics completed (M=3 for each implementation group) is more similar across implementation group than would be expected in a population in which students in all grades participated in System 44 for equal amounts of time.