The California Department of Education's (CDE) contract for the development and scoring of the California Standards Tests (CST) is a multimillion dollar agreement. But all the money seems to go toward developing and scoring the tests themselves. It seems a shame they couldn't carve out some part of the budget to put together a nice little Schoolhouse Rock-type bit. You know, puffy animation and an infectious jingle. Who can forget such classics as "Conjunction Junction" and "I'm Just a Bill (on Capitol Hill)"?The latter would be an excellent model for describing the process by which a question (or "item") finds its way onto a live administration ("operational form") of a CST. Since it appears neither the CDE nor its contractor, Educational Testing Service (ETS) are willing/able to take on this challenge, I will. Not with the animation, jingle, or even a rhyme scheme one would expect from a real Schoolhouse Rock vignette. Just plain old prose.
1. ETS distributes California's content standards to its roster of item writers and solicits items that closely match individual standards.
2. Items submitted to ETS undergo content scrutiny and standards alignment at ETS. Item writers are paid for each item that passes ETS' screening process.
3. ETS sorts the questions by content standards, edits the language and prepares art as needed for the items.
4. ETS presents these questions to the CDE. The CDE convenes a special panel for the specific purpose of reviewing these assessment items, The Assessment Review Panel (ARP). The ARP for science includes professional scientists and science educators (primary through university-level).
5. The ARP evaluates each item for content and alignment to standards. Some items are accepted as is, some are rejected out of hand. Others are sent back to ETS for revision. Those items will have to pass through the ARP before they could move forward.
6. Items that pass ETS/CDE/ARP muster are then placed on operational forms as field test items. Field test items are blended in with operational items on the administered tests. They don't count toward student performance scores. But their inclusion allows for the generation of reliable psychometric data. The psychometric data includes the percentage of students who marked the correct answer, the ability of the item to discriminate between higher- and lower- performing students, and how well the item aligns in the Rasch Model of Item Response Theory.
7. ETS evaluates the performance of each item based on the psychometric data it generated in the field test. Items that perform very well are placed in the bank of questions eligible for use on future operational forms.
8. Items that don't perform well are not immediately discarded. They are submitted to a second meeting of the CDE's ARP. With the psychometric statistics in hand, the ARP evaluates the item again. Some items are given approval to go into the bank of potential operational items. Some are rejected in light of the statistics and unforeseen problems with the item. Some are sent back to ETS for revision. Those items will have to pass through the ARP and field testing before they could be used.
Does such an arduous filtration process guarantee that only perfect items are allowed onto operational forms?
Every bit as much as the process for The Bill on Capitol Hill guarantees only perfect laws are passed.
UPDATE: If you prefer to see a process flowchart without my charm and nostalgia, here it is.
Dean, as always, thanks for the flow of info.
ReplyDeleteWhile the "process" of fielding an item seems robust, the standards on which some tests are based is questionable. In particular, many schools are trapped into offering "Integrated" Science to their under-performing freshman yet, the items on the CST for Integrated Science are simply a random sample of items from the full-blown tests for physics, chemistry, biology, and earth science. Thus an under-performing freshman is expected to answer questions designed for 4 different classes each of which spent a whole "CST year" on one subject. My conclusions: (1) EDT is ripping off the students and tax payers of California because they are too lazy to generate appropriate questions, and (2) the state is guilty of complicity for not providing a sensible alternative for these freshman. At least that's how it looks from my seat.
The Integrated Science CSTs are aligned to the Integrated Science standards as published. And those are "mash-ups" of the Bio/Life Sci, Chem, Earth Sci, and Phys standards.
ReplyDeleteThere will be no "easy" science for low-performing students to take shelter in. The state wants all 9-11 graders to be in a rigorous science class.
My sense is that the state would not shed a tear if your school abandoned Integrated science and placed those students in one of the "pure" science subjects. There are no patchworks of standards or asessment items in those courses.
You identify, people always make comments when anything is predicted to come to pass in 2012, like “obviously that is if the age is still here.” You do achieve that the Mayans predict the world will finish on Dec. 21 (or 23rd)? So in all good chance if anything is accepted to encounter in 2012 there is only the slimmest conceivability that the world hand down have ended already it happens.
ReplyDelete[url=http://2012earth.net/planet_nibiru_2012.html
]Light Beings
[/url] - some truth about 2012