In August, 2003, the California Department of Education (CDE) and Educational Testing Service (ETS) sought to settle on a final form of the Physics Reference Sheet to be included in the California Standards Test (CST) in 9-12 Physics. The CSTs were still relatively new, and the reference sheet used for the 2001—2003 administrations had not been completely ironed out. (I think they were using the reference sheet included on the Physics Bowl Exam, but memory is not always to be relied upon.)
A series of drafts and conference calls ensued. I had been newly appointed to the State Board of Education's Content Review Panel (CRP) for Science and volunteered to take part in the collaboration. There was a representative from CDE, one from ETS, and my CRP colleague, Charles Munger, Jr., and perhaps one or two others in on the process. The debate among the parties was more vigorous than a physics reference sheet might seem to merit. But compromise and consensus prevailed; we settled on a document that CDE and ETS implemented. No further work on the document was ever called for or done.
In 2011, ETS changed the reference sheet. A colleague alerted me to these changes and I placed an inquiry to CDE. I got a response from ETS, the contractor in charge of our CSTs. ETS tells me that these changes were needed to accommodate suggestions from the CRP (which is now the ARP: Assessment Review Panel), and to align the 9-12 Physics reference sheet to the Grade 8 reference sheet.
Let's see the changes.
Overview: The Reference Sheet (Click to enlarge)
2004-2011
|
2012
|
 |  |
1. Motion Details
2004-2011
|
2012
|
 |  |
2. Force Details
2004-2011
|
2012
|
 |  |
3. Energy and Momentum Details
2004-2011
|
2012
|
 |  |
 |  |
You might like the changes; you might dislike the changes. Arguments can easily be made on either side. And those arguments were made in August, 2003!
The details are important, but they're not my primary concern.
I hadn't so much as looked at the reference sheet since 2008, because the matter was settled in 2003. I failed to imagine that while there was no money to continue the Released Test Question process mandated by the STAR test legislation, there was money available to alter the reference sheet. I wouldn't have known about the changes at all if not for an inquiry sent to me by a colleague. (I was incredulous until they included an image.)
As a member of the Assessment Review Panel, I do not recall any suggestion that the reference sheet be modified. I know the ARP never reviewed or approved the modifications.
Having served on the CRP/ARP all these years, I have been made to appreciate the value and importance of the psychometric analysis that is done on the tests and the test items. Psychometric analysis places demands on year-to-year consistency. CRP/ARP suggestions for changes to the test were routinely waved off by ETS/CDE because such changes would break the integrity of the psychometrics.
Altering the reference sheet flies in the face of psychometric integrity. It may be that the CST is so close to sunsetting that psychometrics no longer valued.
So my concern is that changes were made to the CST's Physics Reference Sheet without the involvement of or notice to the Assessment Review Panel. I don't think the reference sheet needed modification in the first place. And given the vigor of the 2003 debate over the original reference sheet, I am troubled that ETS unilaterally modified the sheet based on their perception of implications of ARP feedback on individual test items.
Every change they made was debated and rejected in the August, 2003 reference sheet authoring process. I worry that a new team of individuals came into the process—not knowing the history of contention and consensus that preceded their involvement—and said, "Hey, you know what this thing needs is..."
Amid all those changes, no one could edit the "1 J = 1 N m" to read "1 J = 1 N·m"? With NGSS looming, it's safe to say the CST will run its entire course in anticipation of a notation change that California physics textbooks have yet to make. But I digress.
In sum: Educational Testing Service unilaterally changed a testing document that was originally built through stakeholder consensus. They did so without asking California's Assessment Review Panel for either permission or approval. And they didn't notify the panel or the public that they had done so. So yeah, I'm not happy.
2 comments:
This incident sure looks like someone with insider influence made some changes that suited them. I am not surprised they kept it quiet. The changes are trivial and a step backwards in my opinion. If I ever am granted insider influence I hope I am able to use it for something more significant.
-Dan Burns
Like the unit product dot!
Yeah, it's like the guy who got fired from the bank when they caught him stealing pencils.
The problem (aside from unnecessarily obfuscating the equations in a manner that doesn't add value to them) is that there will always be new people coming on board at ETS. Every new team member with any kind of physics background is going to look at the extant reference sheet and say, "You know, what this thing needs is..."
The collaborative / contentious / consensus process allowed us to consider the pros and cons of all the variants. What is being used now was rejected in 2003. It was resurrected by someone not present in the original process—someone unaware of why it was rejected in the first place.
Post a Comment