With this bulletin, OSDE seeks to clarify information about elements of the SY2024 public school report card that have elicited questions from several of our educational stakeholders. The SY2024 report card introduces:
-
New indicator model subtypes that allow OSDE to provide more schools with an overall A-F Report Card grade, and;
- Use of concorded scores for comparing the results of the 2023 and 2024 OSTP ELA and mathematics tests in consideration of revised standards.
New Report Card Indicator Model Subtypes
The Oklahoma Public School Report Card is expected to include all Oklahoma public schools in its system of annual, meaningful differentiation (aka, school accountability system). This ensures that schools with groups of students performing significantly below expectations receive additional support to improve their educational systems in the expectation that making targeted changes can lead to improvements in student outcomes and teaching pedagogy.
When OSDE designed its accountability system in 2016 in response to the Every Student Succeeds Act (ESSA), it determined that grades would be calculated for an indicator only if 25 or more student results were available. If not, it would not report an indicator grade for the school. Schools that do not have a grade for key indicators do not receive an overall report card grade. Since Oklahoma has many small schools, this approach excluded a significant number of schools from the opportunity to be designated to receive supplemental support for school improvement. This initial approach fell short of the requirements for a state accountability system. Considering this, OSDE has engaged in developing incremental improvements to meet ESSA's expectations.
For SY 2023, to calculate indicator grades for smaller schools, OSDE implemented a modified approach that pooled up to three years of student data to find at least 25 student results upon which it could calculate an indicator grade. This three-year rolling calculation method is called the Small School Indicator model. When this method is used, report card users will notice a small "SS" beside the indicator.
Still smaller schools had too few student results for some indicators to receive an overall grade in the accountability system. Hence, for 2024, leveraging the methods of the Small School Indicator model, OSDE devised the Tiny School Indicator model, which looks back three years to find at least 10 values upon which to calculate the indicator grade. By policy, 10 student results are the smallest number for use in reporting. Indicators calculated using this method are indicated by a "TS" symbol beside the indicator.
OSDE observed that some schools were still not able to receive an overall grade in the accountability system. These are schools with a grade span that does not allow for the direct calculation of the required Academic Achievement, Academic Growth, and Graduation Rate indicators, e.g., PK-3 schools, 9th grade centers, etc. To ensure that these schools are not excluded from opportunities for supplemental support, OSDE developed the "Feeder School" model, which calculates indicators based on the performance of the schools that students attend upon leaving the feeder school. Indicators calculated using this method are indicated by an "FS" symbol beside the indicator.
The advantage of implementing these model subtypes is that now almost every public school in Oklahoma has an overall report card grade based on key performance metrics, providing them an opportunity to receive supplemental support if warranted.
Users should note some complications of these approaches that OSDE attempted to avoid in its initial (2016) accountability system design:
1) Small, tiny, and feeder school indicator model subtypes do not provide a snapshot of the performance of enrolled students for a single year. Therefore, when schools with small student populations implement changes that lead to improved outcomes, the impact of those changes will take time to show up because the performance of students prior to the change dilutes the effect in the data. Likewise, emerging challenges also take time to reveal themselves.
2) Particularly when the tiny school indicator method is used, the performance of one or two students matter. One exceptional student performance (positive or negative) makes a bigger splash in a group of 10 than it does in a group of 100. Statisticians refer to data sets with lots of variability as "noisy" data. Noisy data is less reliable than that derived from a larger population sample. Measuring over a longer time span to get a bigger sample helps (hence why these model subtypes pool data for up to three years) but does not eliminate this issue. For tiny school indicators, this is compounded in that the exceptional performances may skew the data for a couple of report card years.
Prudent report card users are advised to take into consideration these issues when evaluating the meaning of a school- or indicator-level letter grade when these methods are used.
Use of Concorded Scores for Academic Achievement and Growth
With the rollout of revised tests in spring 2024, aligned to subject standards revised in 2021 (ELA) and 2022 (mathematics), OSDE determined that to accurately calculate academic growth it was necessary to equate 2024 test scores to equivalent 2023 values. OSDE sought expert guidance from its Technical Advisor Board and testing provider (Cognia) to make this determination. Using the concordance tables provided by Cognia, which were developed using methods approved by the Technical Advisory Committee, OSDE equated SY24 test scores to their SY23 scale score equivalents. This also allowed OSDE to maintain the timelines for improvement toward proficiency targets established for those subjects.
For context, OSDE established revised timelines for achieving long-term goals for student proficiency for several disadvantaged groups in its 2022 ESSA amendment as it rebooted its assessment and accountability systems post-pandemic, and all student groups are expected to achieve the long-term goal of proficiency based on a score of 300 by 2032. Because the scale score targets were developed on the basis of student performance on pre-2024 test results, the annual targets are not affected by equating scores to previous standards. The equated scores are consistent with the annual scale score targets published in Appendix A of Oklahoma’s Consolidated State Plan.
For ELA and mathematics, 2024 tests create a new benchmark year (akin to 2021). Thus, OSDE will consult with its Technical Advisory Committee as it reviews student performance on the revised tests and determine whether the scale score targets for some groups need to be revised.
|
