Joan L. Herman
Center for Research on Evaluation, Standards, and Student Testing (CRESST)
This report provides an external evaluation of the extent to which the PALS project has met the goals set forth in its proposal. The report is based on project efforts through September 2000 and will be updated at the end of the project in December 2000. The evaluation focuses on the quality, implementation, and impact of the assessment resources. The role of the external evaluation has been to oversee the project evaluation design, provide direction for the formative evaluation activities conducted by the PALS project team, and appraise progress. For each of the three years of the project, the external evaluator has appraised the assessment collection efforts, reviewed data collection instruments and analyses, and specified additional analyses that might be warranted. In addition, the external evaluator prepared the evaluation section of the annual progress reports and presented evaluation findings and recommendations at the annual meetings of the PALS Steering Committee. Project evaluation activities have focused on two areas: (1) documenting the quality of assessments and supporting resources placed on the PALS Web site, and (2) evaluating the processes and effectiveness of various approaches to implementation.
Development of Online Assessment Resources
The Assessment Collection. The PALS project has met and exceeded its goals for obtaining science performance assessments contributed by a range of assessment development programs. The Professional Development Pool currently offers 170 tasks, with another 50 tasks being processed. The project expects to have posted over 225 K-12 tasks by the end of the funding period, therefore exceeding the original goal of 150 tasks. At the elementary, middle and high school levels, the project has augmented the range of NSES content areas represented, importantly adding tasks in life sciences, earth/space science and technology. Personal and Social Perspectives are still very spottily represented, but as previously suggested, those interested in assessing this content area may be able to do so by augmenting an existing prompt to add a question representing the perspective domain. History and Nature of science tasks (as a standard of primary focus) are non-existent, which seems to represent the priority assessment programs are according this domain and/or that performance tasks are really not needed to assess it.
States, districts, and test publishers have been hesitant to participate in the proposed PALS Accountability Pool for high stakes testing purposes. They appear concerned about the confidentiality and security of the Web. The CCSSO Science SCASS is a founding partner of the PALS project, but its state partners have elected to keep the secure science items developed by the collaborative off the Web and available only on CD or in print. Given the reservations about Web security of potential PALS Accountability Pool users, the PALS Steering Committee recommended that the PALS project staff focus on stocking the Professional Development Pool.
The project has had difficulty obtaining complete technical data on the performance tasks it is collecting. The difficulty in many ways underscores the importance of the PALS initiative and providing a permanent, accessible repository for performance assessments. Performance tasks are essentially fugitive documents that are difficult to access and often not formally documented in any formal reports. Instead, try-out and field test data languish in the files of administrators and developers who were responsible for the original development. By the time tasks are retired and released, memory fades and field test data files become more and more difficult to retrieve. Furthermore, as assessment program administrators and developers change positions and move from one state or district to another, the historical memory is essentially lost. It is difficult to just be able to retrieve the tasks themselves and corresponding rubrics and student work; searching for complete technical data (even if it ever existed) is too often beyond the commitment of the assessment resource providers. As a result, the technical data on the PALS tasks is incomplete.
To attempt to rectify this situation as much as possible, the evaluation has developed a follow-up interview protocol to try to solicit available technical information and to assure that all tasks have been developed through a process that included field testing and rigorous content and bias reviews. The PALS staff has used the protocols to interview assessment program directors of the states of Connecticut, Kentucky, and New York. Technical information has also been documented for tasks developed by the CCSSO Science SCASS, the New Standards Science program, and the Illinois Science Teachers Association. The PALS project is currently posting this technical quality information on the PALS Web site. This information makes PALS a unique and important resource in providing performance assessments of documented technical quality.
In practice, however, it may be that detailed technical data are not as important for the professional development uses of PALS. Here teachers, schools, districts, etc. may only need to be assured that the tasks have undergone serious content and bias reviews to assure that the assessments represent good science and will not be unfair or politically explosive. Information that the tasks and rubrics have been field tested offers additional assurance that the assessments have elicited intended student responses and that teachers have used the scoring rubrics with consistency.
Task-by-task technical data would be important for an accountability pool that might be used for high stakes purposes, but it seems apparent that districts and states are hesitant to participate at this level. They appear concerned about the confidentiality and security of the Web. In any event, any use of PALS' tasks for a specific accountability purpose would require new field-testing and analysis of technical data from the target population and assessment context.
The PALS Guide. The PALS project has developed the PALS Guide--a set of background materials, guidelines, and examples to support adaptation and development of the performance assessments. This addition to the online resources goes beyond the proposed development activities and represents the project team's responsiveness to feedback from professional development coordinators for such supporting materials. The Guide incorporates materials and procedures drawn from a range of widely used performance assessment resources. The Guide is an important contribution to supporting the use of PALS. Moreover, the Guide represents a significant, promising move to online professional development for improving assessment practice.
Science Standards. A critical component of PALS is its support for linking assessments in the online collection to the science standards that the tasks are intended to test. With the press for standards-based reform, this feature of PALS is an important tool for aligning standards and assessments. The PALS Web site allows users to search for assessments indexed to the National Science Education Standards (NSES). In response to needs expressed in professional development sessions, the PALS project has gone beyond its intended scope of work to add science standards from states, e.g., Texas and Illinois, to illustrate the possibilities for states and districts to customize searches for PALS assessments by allowing their teachers' to search the assessment collection for tasks indexed to state or local science standards. This feature, plus the assessment planning charts that graphically represent the indexing of targeted science standards to assessment tasks selected by users, are unique and strong tools for supporting integrated science reform programs.
Curriculum Frameworks. The PALS project has created another tool to support standards-based science reform efforts. For a few widely-used science reform curricula, FOSS and STC, the project has posted tables that illustrate the capacity of the PALS system architecture to index modules and units to the NSES and to relevant PALS assessment tasks. The potential for cross-indexing science curriculum units, standards, and assessments holds great promise for supporting science reform.
PALS Technical Architecture
The PALS project has devoted significant effort to the design and improvement of the technical architecture. The relational database allows users to easily search by standards or by grade and topic. A new function will allow users to search for PALS tasks relevant to curriculum modules if the program framework has been posted on the Web site. The architecture allows dynamic generation of assessment charts to support assessment planning by displaying the PALS tasks indexed to science standards selected by the user. The numerous studies conducted by the project with individual users have informed the design and redesign of the interface. The user studies resulted in refinement of the graphical-user interface, which, in its present form, has been judged easy to navigate. In addition to the core functions for supporting task selection and site navigation, the new features including threaded discussion boards, personalized assessment charts (My Chart), user ratings and discussion of individual tasks, searching by other science standards and curriculum frameworks, and the PALS Guide. These features have resulted in a PALS Web site that offers far more functions and features than other assessment Web sites. Moreover, by placing the PALS server behind a firewall and planning to employ Secure Sockets Layer, the industry-wide standard for encrypting data transferred across the Internet, the PALS project can address the major security concerns raised by potential users of the Accountability Pool. In sum, the PALS architecture continues to push the envelope of the state of the art and is a leader in on-line assessment sites.
PALS Implementation
A PALS project goal has been to study models for implementing the online resources. Since online educational resources were relatively new when the PALS project began in 1998, the project has had to develop approaches for promoting awareness and use. Unlike print materials that are distributed at teacher workshops or by publishers, the PALS resources are electronic. The PALS project has augmented traditional print and presentation outreach strategies with dissemination via the collection's online Web site and by links to and from related Web sites and online professional development discussion groups in SRI's TAPPED IN. The PALS project has been very active in raising awareness of PALS and trying to solicit extended implementation sites. The project has vigorously pursued implementation through outreach and dissemination activities and project-supported professional development sessions as evidenced in Table 2 of the draft final report by the many presentations to professional organizations and federal, state, and local education agencies.
PALS Impacts
The PALS project evaluates its impacts in several ways: (1) Web traffic, (2) user feedback, (3) implementation, and (4) quality and utility judgments by educators. The number of visitors to PALS is very promising, as is the international distribution of educators viewing the site. Reactions to questionnaires distributed at eight PALS demonstrations show positive findings. Teachers and administrators find PALS generally easy to use, plan to use it, and point to a variety of uses they anticipate. Most popularly, participants planned to use PALS tasks for classroom assessments and using them to work with other teachers to help them understand performance assessments and how to use them (See appendix for data summary). Nonetheless, despite many promising meetings and tentative agreements, the PALS project has had difficulty studying extended professional implementation efforts, raising questions about the PALS niche and how to appropriately position it for effective use. In that regard, PALS staff is following-up with workshop participants to see whether and how they're actually using PALS, and if not, why. The study currently underway of 100 teachers at a Texas year-long PALS implementation site will provide more information on the role PALS can play in supporting face-to-face professional development on assessment. Such data will be used to explore marketing options and to fine-tune the PALS niche.
PALS is becoming a highly regarded assessment resource in the field, as evidenced by the number and range of agencies and organizations that have written enthusiastic letters of support and commitment for the PALS project and the further development of its professional development and technical assistance services. These organizations and agencies include: (1) key professional organizations-the National Science Teachers Association (NSTA), the National Education Association (NEA), the ERIC Clearinghouse on Assessment and Evaluation, (2) NSF-funded science implementation centers, (3) state education departments; and numerous science curriculum projects.
Project Maintenance and Sustainability
As a research and development project, the PALS project has committed most of its resources to building the content and technical architecture of the Web site. At the outset, the accountability pool seemed to have the greatest potential for revenue generation. However the reservations of testing programs about Web security has turned the PALS project to other business models. The production of the PALS CD is promising. Yoking the PALS online resources to other professional development programs and face-to-face models offers another possibility. Furthermore, technical assistance services through the custom curriculum indexing feature offers a potential revenue stream to support and maintain the project.
Summary
In summary, PALS has surpassed its goals in developing an on-line resource of performance assessments in science. It is unique in bringing together state of the art knowledge of assessment with state of the art technology to provide an on-line resource that is very unique. Users have been very positive about the usability, quality and utility of PAL and see a variety of applications in their schools and classrooms. Their enthusiasm is echoed in the number of states, districts and other agencies who are committed to further implementation. Research suggests that teachers and schools really need the kinds of resources that PALS provide to bring the vision of standards-based science reform to fruition. The examples, support, and concrete tools that PALS makes available should be invaluable in helping schools and teachers to understand the meaning of standards and to build schools' and teachers' to assess and improve.