Experimental Research on Preschool Learning Technology in Rural Utah
Part 1: Our Research Design
Utah Preparing Students Today for a Rewarding Tomorrow (UPSTART) is an in-home interactive computer-based pre-school program created by the Waterford Institute using Waterford Early Learning software. The program contains two strands of curriculum for early learners: reading (phonological awareness, comprehension/vocabulary, and language concepts) and math/science (numbers and operation, measurement, and earth and life sciences).
In 2014, the Waterford Institute was awarded a federal Investing in Innovation (i3) validation grant to expand UPSTART to families residing in rural districts in Utah and the Evaluation and Training Institute was contracted to conduct a systematic evaluation of the program’s effectiveness.
About the Waterford Early Learning Reading Software
The UPSTART reading curriculum is designed to promote the development of beginning literacy skills[1] that will prepare young children for entry into school by adapting to each child’s skill level, and providing an individualized learning experience. The reading skills taught by the Waterford Early Learning Program at Level 1 of the curriculum include:
Phonological Awareness: phonemic segmenting and blending
Phonics: letter name knowledge, sound knowledge, and word reading
Comprehension and Vocabulary: vocabulary knowledge
Language Concepts: oral reading fluency
At pre-Kindergarten, the recommended program use is 15 minutes a day, 5 days a week, with an expectation that the child will complete at least 1,000 minutes prior to entering Kindergarten.
About the Evaluation Design
We used a randomized control trial experimental research design to study the effects of the UPSTART program on preschool participants’ beginning literacy skills.
Our primary research question is as follows:
Do four-year-old children randomly assigned to receive the UPSTART reading program have higher scores on measures of literacy after one year than four-year children who were assigned to the UPSTART math/science program?
Participants
We randomly assigned families from 15 rural districts in Utah that were using the UPSTART program to one of two experimental groups: Reading or Math/Science. The children in the Reading group received the Reading curriculum and served as our treatment group, while the children assigned to the Math curriculum acted as our control group. Both groups of children logged into the same UPSTART software platform, but they received different curriculum depending on their condition.
Before randomly assigning children to the two experimental conditions, we administered a pre-test battery of early literacy measures in the Summer of 2014 to establish their baseline literacy skills before they began using the UPSTART program. Students participated in the UPSTART program during the 2014-15 school year and after completion of the program we post-tested both treatment and control children in Summer 2015 (before they began Kindergarten). Our experimental design is diagrammed as follows:
Evaluation Design – Preschool Study
Instrumentation
We measured early literacy with two standardized instruments, the Preschool Early Literacy Indicator (PELI) and the Brigance Inventory of Early Development (IED-III). We also assessed math achievement with the Brigance IED.
The Preschool Early Literacy Indicator (PELI) is a storybook-embedded assessment of essential pre-literacy and oral language skills. The assessment is designed for preschool and pre-kindergarten students (ages 3-5) and measures the following constructs:
Alphabet Knowledge
Vocabulary and Oral Language
Phonemic Awareness
Comprehension
The Brigance Inventory of Early Development III (Brigance) was selected as an early literacy measure of phonics and vocabulary knowledge and as a measure of pre-kindergarten academic and cognitive skills. Nine Brigance literacy scales will be administered in the academic skills /cognitive domains of literacy:
Experience with Books and Text
Recites Alphabet
Visual Discrimination
Uppercase Letter Knowledge
Phonological Awareness
Auditory Discrimination
Phoneme Manipulation
Survival Sight Words
Basic Pre-Primer Vocabulary
Initial Findings
In order to ensure that the treatment and control groups were statistically equivalent before beginning the UPSTART program, we conducted a preliminary series of t-tests on pre-test scores between the two groups to identify any significant pre-existing differences on the outcome variables of interest.
There were no significant pre-program differences between preschoolers who were assigned to UPSTART reading and those who were assigned to UPSTART math on the Brigance Literacy Composite Score. Moreover, preschoolers were equivalent on all eight subtests of the Brigance Literacy measure. Similarly, results from pre-test data indicate that the treatment and control group children were essentially equivalent on the PELI Composite, as well as on all the subtests.
ETI will be presenting our research findings over the next few months, including results from literacy post-tests upon completion of the UPSTART program, so subscribe to our Twitter feed to receive updates.
For more information on the i3 Rural UPSTART evaluation, click here.
[1] The UPSTART program is designed to promote reading (literacy), but also has curriculum developed to support Math and Science learning. We have operationally defined “school readiness” in terms of literacy; however, Math and Science are core skills for success in elementary school (and beyond).
