There is a great annual, intellectual sport of criticizing the released items on California's STAR test in physics. Each year, the California Department of Education releases 25% of the previous year's test. When the new items are published, the listservs light up with hue and cry: many (if not all) the items are deemed trivial, poorly constructed, or flat-out wrong. Or all of the above!
Some of the criticism is well-founded, some not so much. I doubt there is a question in existence that would be unanimously approved by a group of 100 physics instructors. Add to that the reservations that many teachers have regarding state-mandated science standards, and you've got a recipe for, well, enthusiastic criticism.
It is in that spirit that I present these questions from Illinois (Word document). The document appears to be one district's attempt to simulate a statewide assessment given to 7th-graders. The state's released test questions can be found here (PDF).
Please look at both and see if you think they constitute an assessment you would hope for in your fair state. How would you assess the writing, grammar, content, and difficulty of these items? Let me know in the comments.
Admonishment and acknowledgments to Freakonomics via Bad Astronomy. Those sites are portaying the school district's practice test as the state's assessment items. I fear they are in error to do so.
UPDATE: Here's an item from the official ISAT RTQs. I would have recommended rejection had it been offered for California's physics STAR test.
What property of light waves can be observed as light waves pass from one medium to another and change speed?
A Diffraction
B Refraction
C Reflection
D Separation
Seems straightforward enough. Why would I have rejected it?
They obviously want refraction. So what's wrong? Two correct answers, to start with, since reflection also can occur at boundaries. Then there's the fact you don't get refraction if the light hits parallel to the normal, and the question doesn't specify that light is hitting at an angle to the normal.
ReplyDeleteI suspect most kids would actually get this kind of question right, since they don't think too hard about the subtleties and just go for the "obvious" correct answer.
I should go ask my IL 7th grader what he thinks the answer is.
Il Rez,
ReplyDeleteYou are correct about the reflection. That's why I would reject it. You're also correct in your prediction that most students (who studied the content in class) would get it right. The question would generate "good stats" and get a green light to go from field testing to operational form.
But refraction DOES occur when light hits perpendicular to the boundary. The propagation does slow down. It doesn't undergo a change in direction, but it is refracted.
The question would have been better if reflection was not among the choices.
The weakness is not earth-shattering, and the sky will not fall. But an easy fix could have improved the item.
I disagree that the definition of refraction is light changing speed when it passes into another medium. Every definition that I have read for refraction states that it is the bending of light. I believe the word preceeds the knowledge of the changing speed of light. Refraction does not occur if the light strikes normal to the surface.
ReplyDeleteAh, yes, the multiple definitions that are almost but not quite compatible problem... My working definition of refraction is the same as Dan's: the bending of light that hits the interface at an angle to the normal, caused by the change in wave speed. If it's just the change in speed, then yeah, angle wouldn't matter. (Nor, I think, would that kind of distinction matter to the kids in any event.)
ReplyDeleteIt does amaze me, though, how often question writers don't recognize that they have multiple correct answers in a multiple choice test. This mistake was more understandable than many I've seen--it was just sloppy thinking and at least reflection is not the same as refraction. I've seen questions where one answer duplicates the wording in the book (or standard), and another answer gives a correct example of the exact same concept. But since it's an example and not the "book" answer, it must be wrong, right? I've also seen question where, if the tested concept is "If A, then B" the question asks "If B, then which of the following?" and gives A as a choice.
some of these questions truly scare me..
ReplyDeletefor example #1 can be paraphrased as follows
I am going to say a word. the word is WATER
which word did I say
a)dirt
b)oil
c)water
d)light
i guess its ok from a very basic reading comprehension question - but this seems more appropriate for maybe a 4th grader?
#23 - wow - i guess they are going for the parachute in d presents less area? But wow - that question is problematic
KBlack,
ReplyDeleteYes, #23 is problematic--I didn't like it, either. I guess they might have field-tested a similar question with all parachutes having the same design and the area being the only variable. But too many respondents got it right. So they toughened it up by adding an extra variable. The geometry of the chute can affect its drag characteristics, so I would have choked on approving that one.
Then again, this all underscores the challenge of writing a good item. Everyone thinks they can do it until they try to. I'd pay a dollar to see an item that could get approval from a room of 100 physics teachers.