The California Department of Education's (CDE) contract for the development and scoring of the California Standards Tests (CST) is a multimillion dollar agreement. But all the money seems to go toward developing and scoring the tests themselves. It seems a shame they couldn't carve out some part of the budget to put together a nice little Schoolhouse Rock-type bit. You know, puffy animation and an infectious jingle. Who can forget such classics as "Conjunction Junction" and "I'm Just a Bill (on Capitol Hill)"?
The latter would be an excellent model for describing the process by which a question (or "item") finds its way onto a live administration ("operational form") of a CST. Since it appears neither the CDE nor its contractor, Educational Testing Service (ETS) are willing/able to take on this challenge, I will. Not with the animation, jingle, or even a rhyme scheme one would expect from a real Schoolhouse Rock vignette. Just plain old prose.
1. ETS distributes California's content standards to its roster of item writers and solicits items that closely match individual standards.
2. Items submitted to ETS undergo content scrutiny and standards alignment at ETS. Item writers are paid for each item that passes ETS' screening process.
3. ETS sorts the questions by content standards, edits the language and prepares art as needed for the items.
4. ETS presents these questions to the CDE. The CDE convenes a special panel for the specific purpose of reviewing these assessment items, The Assessment Review Panel (ARP). The ARP for science includes professional scientists and science educators (primary through university-level).
5. The ARP evaluates each item for content and alignment to standards. Some items are accepted as is, some are rejected out of hand. Others are sent back to ETS for revision. Those items will have to pass through the ARP before they could move forward.
6. Items that pass ETS/CDE/ARP muster are then placed on operational forms as field test items. Field test items are blended in with operational items on the administered tests. They don't count toward student performance scores. But their inclusion allows for the generation of reliable psychometric data. The psychometric data includes the percentage of students who marked the correct answer, the ability of the item to discriminate between higher- and lower- performing students, and how well the item aligns in the Rasch Model of Item Response Theory.
7. ETS evaluates the performance of each item based on the psychometric data it generated in the field test. Items that perform very well are placed in the bank of questions eligible for use on future operational forms.
8. Items that don't perform well are not immediately discarded. They are submitted to a second meeting of the CDE's ARP. With the psychometric statistics in hand, the ARP evaluates the item again. Some items are given approval to go into the bank of potential operational items. Some are rejected in light of the statistics and unforeseen problems with the item. Some are sent back to ETS for revision. Those items will have to pass through the ARP and field testing before they could be used.
Does such an arduous filtration process guarantee that only perfect items are allowed onto operational forms?
Every bit as much as the process for The Bill on Capitol Hill guarantees only perfect laws are passed.
UPDATE: If you prefer to see a process flowchart without my charm and nostalgia, here it is.