|
|
|
| Home | Site Index | Help | FAQ | REGISTER | ||
|
|
Technical Report 2
|
||||
| The performance measurement system will consist of child and family outcomes for different child ages within the early childhood age range as well as indicators and sources of data corresponding to each outcome. These child and family outcomes, indicators, and sources of data must be useful for tracking the progress of a broad range of children and families with different disabilities and characteristics and for measuring the impact and effectiveness of early childhood programs (RFP 84.024S[3]). |
In response to this request, investigators at the three universities proposed to develop two sets of outcomes: (a) common competencies applicable to all children between birth and age eight, and their families; and (b) a set of subgroup specific outcomes for any subsets of children and families for whom the general outcomes did not describe development adequately.
In late October, 1996, soon after the Institute began formal implementation of the proposal, all of the principal investigators gathered in Minneapolis/St. Paul, MN to review work scope and plan strategy. We met with Drs. Stanley Deno and James Ysseldyke, two researchers at the University of Minnesota, who reviewed the theoretical orientation of the Institute and its initial goals. As a result of these meetings, as well as contact between principal investigators and staff at OSERS, we arrived at two conclusions. First, given the substantial body of extant theoretical and empirical work on developmental outcomes for young children and their families, we concluded any new analyses would likely replicate previous ones. We decided to rely on existing sources of knowledge to begin selecting and validating outcomes as the foundation for development of a comprehensive measurement system. Second, we initially proposed to bring individuals together across early childhood constituency groups to select general growth outcomes using consensus-building techniques. However, we discovered previous consensus-building efforts on early childhood outcomes across a variety of “stakeholder” groups yielded high levels of agreement, regardless of group composition, geographical location, or cultural perspective. Thus, efforts to organize meetings between constituencies to build consensus on such outcomes seemed redundant and a poor use of our valuable resources. Instead, we turned our attention to a comprehensive review and synthesis of existing outcome statements.
We set the following criteria for selecting prospective outcome statements for children between W birth and age eight: (a) individualized focus; (b) comprehensiveness across developmental domains; and (c) continuity across age.
Most existing outcome systems for young children focus on aggregated results for groups of children (see Technical Report #1: Accountability Systems for Children Between Birth and Age Eight for a more detailed discussion of outcome systems for young children). For example, to measure progress toward Goal 1 of the National Education Goals (i.e., By the year 2000, all children in America will start school ready to learn), governmental agencies generally report nomothetic statistics, such as the percentage of children born with one or more health risks or the percentage of preschoolers whose parents read to them (Federal Interagency Forum on Child and Family Statistics, 1997; National Education Goals Panel, 1995). Likewise, the National Center on Educational Outcomes (NCEO) recommends using frequencies of children and families who demonstrate certain practices or skills (e.g., percent of children who initiate and follow through on activities) or who participate in certain activities and environments (e.g., percent of children enrolled in early care and education programs) as the sole criteria for establishing indicators of outcomes for young children (Ysseldyke, Thurlow, & Gilman, 1993a; Ysseldyke, Thurlow, & Gilman, 1993b).
While nomothetic statistics provide important information about early childhood programs, they do not help educators evaluate an individual child’s developmental trajectory nor begin to fashion interventions specific to the child’s needs if her trajectory fails to meet expectations. In response, we planned to select and refine outcomes to describe individual children’s growth and development. By focusing on individual children, we believe these outcomes will serve as the foundation for creation of indicators early childhood educators can use to track the developmental progress of individual children, especially children with disabilities, as well as drive creation of additional solutions-oriented assessment tools for educators to employ when an individual child’s progress fails to meet predetermined expectations. Thus, an essential feature of our focus on individualized outcomes rests on the consequential validity of these statements.
From the outset, we planned to include outcomes across the range of developmental skills typically expected of children between birth and age eight. These include skills in language and communication, cognitive, motor, social, and adaptive (or, activities of daily living) domains. While we expected to include outcomes across all of these developmental domains, we did not necessarily plan on labeling outcomes using these traditional domain headings, focusing more on the functional aspects of statements.
We also hoped to use a common set of outcomes across the entire age range of children upon whom the Institute will focus, i.e., birth to age eight. We realized specific milestones for an infant or toddler will differ markedly from skills expected of a five-year-old or eight-year-old child. For example, we expect most preschoolers will produce words and sentences as an outcome of their communication development (Brown, 1973; Wells, 1985), in contrast to preverbal infants who rely more on joint attention, gestures, and sounds to communicate (Wetherby & Prizant, 1992). However, we believed it possible to maintain the functional continuity of outcomes within each developmental domain and across age to a significant extent, allowing Institute staff and others to apply an initial set of broad outcomes to all children between birth and age eight. Thus, while infants, preschoolers, and early elementary-aged children may engage in different behaviors to communicate, in both cases they indicate their wants and needs and express meaning to others.
In light of these interests, we decided to pursue the following action steps to generate an initial set of outcomes. First, we would identify sets of prospective outcomes by examining existing resources, focusing on reviews of literature and extant assessment instruments. Given the volume of resources on the topic of developmental outcomes in early childhood, the relative consensus across early childhood “stakeholders,” and the accumulated experience of the Institute’s principal investigators in the field of early childhood assessment and intervention, we believed these initial efforts could be based on a relatively targeted set of resources.Second, we wanted to select a comprehensive but parsimonious list of outcomes for each of three age groups (birth-two-year-olds, three-to-five-year-olds, and five-to-eight-year-olds), paying particular attention to the terminal ages of these groups (i.e., children at ages three, five, and eight). We intended to accomplish this step by aggregating statements of developmental skills from existing resources and following an inductive process to determine broader outcomes, using constant comparative methods described in Lincoln and Guba (1985).
Third, we planned to review broad outcomes developed within each age group for their applicability across the three age groups. Staff at each Institute site would create a table of outcomes appropriate for children within their assigned age range, and then review this table with the two other sites. In reviewing these tables, we planned on paying particular attention to the consequential and social validity of prospective outcomes across all children between birth and age eight. That is, we felt two of the most important standards to heed centered on: (a) the impact of outcomes on understanding and enhancing developmental trajectories of young children with and without disabilities; and (b) how early childhood constituencies would perceive the appropriateness and usefulness of these outcomes. We realized future use of the general growth outcomes would hinge greatly on acceptance by families, teachers, administrators, and other individuals involved in early childhood education.
To assess the extent to which our outcomes were indeed acceptable to parents, teachers, and others, we planned to conduct a national survey of parents and early childhood/early elementary professionals, asking them to rate each of the general growth outcomes and provide revised outcomes if they wished. Results from this survey (described in Technical Report #3: National Survey to Validate General Growth Outcomes for Children Between Birth and Age Eight) generated validating feedback on selected outcomes, as well as provided information on the need to adapt outcomes for specific subgroups of young children.
In general, staff at all Institute sites used the guiding principles described above for the selection of I general growth outcomes. However, each site approached the task in slightly different ways. This section details how staff at each site derived initial outcome statements for their particular age group.
Staff at the Juniper Gardens Children’s Project, University of Kansas, developed outcomes for children between birth and age three. At first, they examined assessment instruments, lists of developmental milestones, early childhood curricula, journal articles, and book chapters. Examples included the Battelle Developmental Inventory (Newborg, Stock, Wnek, Guidubaldi, & Svinicki, 1984), AEPS Measurement for Three to Six Years (Bricker & Pretti-Frontczak, 1996), Developmentally Appropriate Practice in Early Childhood Programs (Revised Edition) (Bredekamp & Copple, 1997), The MacArthur Communicative Development Inventories (Fenson et al., 1993), and From First Words to Grammar: Individual Differences and Dissociable Mechanisms (Bates, Bretherton, & Snyder, 1988).1
Using developmental “areas” formulated by the authors of the Battelle Developmental Inventory (Newborg et al., 1984) (e.g., peer/adult interaction and self-concept within the Personal-Social domain), Juniper Gardens staff assigned each of the skills they found to one or more of these areas. Once they had assigned all skills to areas, they discussed ways of converting area labels into outcome statements. They focused especially on areas that met our pre-determined criterion of consequential validity. Once they decided on an outcome statement, they strove to draft it in terms understandable to a broad range of consumers, including families.
As Juniper Gardens staff developed lists of skills, areas, and domains, they noticed they could channel their work into two different directions. They could draft outcomes to represent what children could perform at the end of their age range (i.e., competencies achievable by three-year-olds, such as using multi-word utterances to express meaning to others) or what children in the entire range of birth to age three could perform in varying degrees (e.g., using gestures, sounds, words, and word combinations to express meaning to others). All project staff discussed this point and decided to focus on outcomes covering the entire age range (a partial attainment model) rather than just the end point (a terminal skill model), emphasizing change in a developmental skill over time (i.e., growth) rather than on the attainment of a particular skill at the end of a sequence. At the same time, though, Juniper Gardens staff paid attention to skills expected of a three-year-old child as a boundary on typical, age-appropriate skills. Staff at the other two project sites adopted this strategy as well.
The Juniper Gardens review of skills and assignment to developmental areas generated a list of 22 outcome statements. Appendix A shows this list of outcomes.
Staff at the University of Minnesota developed outcomes for children between three- and five-years-old. In similar fashion as Juniper Gardens staff, they conducted focused reviews of assessment materials and other literature to identify existing statements of age-appropriate outcomes. They divided resources into two kinds: (a) “known resources,” including many of the same instruments used by Juniper Gardens staff, as well as documents produced by the National Center on Educational Outcomes (Seppanen, Schaeffer, & Julian, 1995; Ysseldyke et al., 1993b) and states with detailed outcomes for young children (e.g., Maryland Commission on the Early Learning Years, 1992); and (b) “found resources,” i.e., journal articles, reports, and related documents discovered through searches of three on-line databases: ERIC, PSYCINFO, and ORBIT (a library database maintained by staff of the National Center on Educational Outcomes at the University of Minnesota). Specifically, they searched for empirical studies, reviews of studies, and statements of expected or desired outcomes by political entities (e.g., state departments of education). They used the following search terms in various combinations to explore these databases: preschool children, outcome, goal, norm, developmental stages, early childhood, child development, pre-kindergarten, and five-year-old. They stipulated such resources must focus on one or more of the five developmental domains of interest (preferably including a comprehensive and detailed discussion of skills within all five domains), be issued since 1977, and published in English. They developed a literature coding form to assist graduate research assistants with their review of a resource’s appropriateness and essential information to gather from such literature.2
Staff at the University of Minnesota combined developmental skills from all resources to generate 819 outcomes, grouped by domain and developmental area (patterned after the process followed by Juniper Gardens staff ). Relying on the constant comparative method described by Lincoln and Guba (1985) [based on their restatement of techniques formulated by Glaser and Strauss (1967)], teams of three individuals evaluated each skill in a developmental area and created categories of outcomes based on similarities between skills. They limited the number of prospective categories per developmental domain, setting a maximum of three. Then, individual staff members redrafted more comprehensive outcome statements, broadening the scope of categories rather than eliminating content from the final set of outcomes. In this way, they wished to maintain parsimony of the ultimate set of outcomes selected while at the same time retain the scope of developmental domains (Lincoln & Guba, 1985).
Next, Minnesota staff conducted a process of team categorizing (Lincoln & Guba, 1985) to reach consensus on the outcome statements fashioned from the original set of 819 skills. At group meetings, they discussed categories in which each person had assigned each of the skills. They let categorical overlap across staff and forced assignment of “miscellaneous” skills to categories drive the final selection of outcome statements within each developmental domain. Finally, they conducted a “member check” (Lincoln & Guba, 1985) on the list of outcome statements by asking each person to repeat the process of assigning skills to categories generated by group consensus, looking especially for skills that could not be placed within a category.
In similar fashion to Juniper Gardens staff, University of Minnesota personnel formulated outcome statements to describe the developmental growth of all children within the age range of three- to five-years-old. They finished the task with a list of 13 outcomes. Appendix A shows this list of outcomes.
The team at the University of Oregon developed outcomes for children between five- and eight-years-old. They initiated their work by consulting a text on child development (Sroufe, Cooper, & DeHart, 1992), as well as many of the same resources used by staff at Juniper Gardens and the University of Minnesota.3 They modeled their work after the process followed by staff at Juniper Gardens, drafting one outcome statement for each selected developmental area within each domain. Their selection of developmental areas depended on project-wide criteria, i.e., whether or not outcomes crafted from developmental areas would be observable, measurable, show child change over time, and be amenable to intervention.
Staff at the University of Oregon fashioned 17 outcomes for children between ages five and eight, heavily weighted toward the cognitive domain. Since this age range represents the period of time when children attend their first three years of elementary school, the emphasis on cognitive (or, academic) skills seemed reasonable. They also asserted the phrase “friendship networks” may encompass the developmental issues of children in this age group more appropriately than “social interaction.” Appendix A shows their list of outcomes.
In June, 1997, the project’s principal investigators from all three sites gathered to discuss work I accomplished at each site on the selection of outcomes and to condense information into a final set. We identified general concerns about the statements fashioned at each site. We considered the vagueness of some statements, the lack of measurability over time in some instances, and the issue of ensuring linkages across the three age groups. We discussed how to present the concept of academic development of children between birth and age five to professionals and families who view this area as the exclusive domain of elementary school and beyond, as well as how to maintain focus on concepts of child development for children between the ages of five and eight.
Investigators also focused on crafting outcomes using language accessible to the widest possible audience, striving to simplify the wording of outcomes without losing substantive information. We partially solved this dilemma by focusing on functional aspects of developmental domains (i.e., a child’s behavior across domains and settings) rather than emphasizing skills in isolation within domains. Thus, while the final set of general growth outcomes somewhat corresponds to a traditional list of five developmental domains in number and basic content, the functional nature of these outcomes differs markedly from tradition.
As an additional result of this analysis, investigators from all three sites noted the commonality across the three age-based sets of general outcomes. Using criteria developed at the beginning of our review (i.e., parsimonious but comprehensive list of outcomes for partial attainment assessment) and those developed during our collective review (i.e., functional, widely accessible language), we derived a composite list of 15 outcome statements that reflected the full content of each age-based effort.
Appendix B shows the five superordinate outcomes and 15 general growth outcomes crafted by investigators to describe all children’s development between birth and age eight. After completing this list, investigators distributed it to project staff and other individuals with no project affiliation to educe feedback on the understandability of outcomes for the widest possible audience. This review process led to drafting two versions of outcomes, one for professionals and one for families. In turn, these two sets of outcomes became the basis of a national survey conducted by project staff, in conjunction with the Minnesota Center for Survey Research at the University of Minnesota, to elicit feedback from a nationally drawn sample of families and professionals in early childhood and early elementary education. Information from this survey helped us validate these general growth outcomes across all children, as well as provided information about the need to adapt outcomes to describe certain subgroups of children for whom these outcomes failed to characterize development adequately (see Technical Report #3: National Survey to Validate General Growth Outcomes for Children Between Birth and Age Eight).
Bates, E., Bretherton, I., & Snyder, L. (1988). From first words to grammar: Individual differences and dissociable mechanisms. Cambridge, UK: Cambridge University Press.
Bredekamp, S., & Copple, C. (Eds.). (1997) Developmentally appropriate practice in early childhood programs (Revised ed.). Washington, D.C.: National Association for the Education of Young Children.
Bricker, D., & Pretti-Frontczak, K. (Eds.). (1996). AEPS measurement for three to six years (Vol. 3). Baltimore, MD: Paul H. Brookes Publishing Co.
Brown, R. (1973). A first language: The early stages. Cambridge, MA: Harvard University Press. Federal Interagency Forum on Child and Family Statistics. (1997). America’s children: Key national indicators of well-being. Washington, D.C.: Author.
Fenson, L., Dale, P. S., Reznick, J. S., Thal, D., Bates, E., Hartung, J. P., Pethick, S., & Reilly, J. S. (1993). The MacArthur Communicative Development Inventories: User’s guide and technical manual. San Diego, CA: Singular Publishing Group.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. Chicago: Aldine. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage Publications, Inc. Maryland Commission on the Early Learning Years. (1992). Laying the foundation for school success: Recommendations for improving early learning programs in Maryland. Baltimore, MD: Maryland State Department of Education.
National Education Goals Panel. (1995). The national education goals report: Building a nation of learners. Washington, D.C.: U.S. Government Printing Office.
Newborg, J., Stock, J. R., Wnek, L., Guidubaldi, J., & Svinicki, J. (1984). Battelle Developmental Inventory. Allen, TX: DLM.
Seppanen, P., Schaeffer, R., & Julian, N. R. (1995). Matching state goals to a model of outcomes and indicators for age 3 (Tech. Rep. 13) . Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes.
Sroufe, L. A., Cooper, R. G., & DeHart, G. (1992). Child development: Its nature and course (2nd ed.). New York: McGraw-Hill, Inc.
Wells, G. (1985). Language development in the pre-school years. Cambridge, UK: Cambridge University Press.
Wetherby, A. M., & Prizant, B. M. (1992). Profiling young children’s communicative competence. In S. F. Warren & J. Reichle (Eds.). Causes and effects in communication and language intervention (Vol. 1, pp. 217-253). Baltimore, MD: Paul H. Brookes Publishing Co.
Ysseldyke, J. E., Thurlow, M. L., & Gilman, C. J. (1993a). Educational outcomes and indicators for early childhood (age 3). Minneapolis: University of Minnesota, National Center on Educational Outcomes.
Ysseldyke, J. E., Thurlow, M. L., & Gilman, C. J. (1993b). Educational outcomes and indicators for early childhood (age 6). Minneapolis: University of Minnesota, National Center on Educational Outcomes.
1 For more information on resources used to develop outcomes, contact the Early Childhood Research Institute on Measuring Growth and Development.
|
