Contextual Constraints in Science Assessment: Effects of Item Sequencing and Item Features on Measures of Student Performance
Loading...
Date
2012-02
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Innovations in automated scoring and related technologies are facilitating increased use of constructed-response (CR) assessment instruments in science education and other content areas. While a large body of work has explored a variety of issues relating to the effects of multiple-choice item sequencing on student performance, comparatively less work has explored these issues using constructed-response assessments. We report on the results from three studies of undergraduate students that examine the effects of item sequencing, difficulty, and surface features on student performance using constructed response items in biology education. Our first study examined the extent to which item sequencing was associated with student performance on isomorphic, constructed-response instruments. In our response corpus, verbosity (number of words) and use of accurate scientific ideas (key concepts) declined significantly from the first to the fourth items. In contrast, use of naïve ideas was not correlated with verbosity or the use of scientific ideas. Our second study examined the interaction between item sequencing and familiarity of items. Use of key concepts in this sample was significantly related to response verbosity and item familiarity, but not item sequencing. The use of naïve ideas was also correlated with familiarity, with more familiar items eliciting more naïve ideas. Our third study examined the interaction between item sequencing and the polarity of item features. In our collection of student responses, verbosity was significantly related to item sequencing but concept use was not. In contrast, the polarity of the item significantly affected the use of key concepts and naïve ideas, corroborating previous research on the gain and loss of traits. Together the results from our three studies identify multiple factors influencing student performance on CR items. Assessments containing items with similar surface features were subject to greater order effects relative to item sequencing that differed in surface features. Additionally, item sequencing had a greater effect on response verbosity, which corresponded with use of accurate scientific ideas. We conclude with a conceptual model accounting for factors influencing student success on CR assessments.
Description
Education and Human Ecology: 2nd Place (The Ohio State University Edward F. Hayes Graduate Research Forum)
Keywords
constructed response, item sequencing, surface features, biology, explanations