A MATLAB Programming Concept Inventory for Assessing First-Year Engineering Courses: a Replication of the Second Computer Science 1 Assessment

Abstract

Validated concept inventories are necessary to accurately gauge student understanding and measure classroom teaching methods. However, the First-Year Engineering (FYE) program at The Ohio State University lacks a validated assessment tool to determine student understanding of MATLAB programming concepts for first-year students. It is critical for FYE programs to have these tools available to allow the program and its instructors to determine the impacts of various teaching styles and curricula.

While there are existing concept inventories available for introductory computer science concepts, these concept inventories are programming language independent and have limitations. One such limitation is that the scores of lower-performing students tend to differ for language-independent versus language-dependent assessments more than those of high-performing students. This indicates that language-independent assessments favor high-performing students. In this study, a new MATLAB-specific concept inventory, the MATLAB Computer Science 1 Assessment (MCS1), is developed by replicating a previously validated foundational computer science concept inventory, SCS1, for use in the FYE program at Ohio State. The goals for this project are to create an isomorphic copy of SCS1 that is specific to MATLAB, conduct think-aloud interviews with first-year engineering students to determine how the assessment is being interpreted, and give both the MCS1 and SCS1 assessments to first-year students. These data will then be used in a future project to validate MCS1.

The result of this study is assisting in successfully developing the MCS1 concept inventory by replicating SCS1. Think-aloud interviews were conducted with six first-year students. The MCS1 and SCS1 assessments were given to a combined total of 724 FYE students in autumn 2019. Preliminary data demonstrates a statistically significant difference between SCS1 and MCS1 scores, indicating that MCS1 cannot be validated against SCS1. As a result, a future study is required to validate MCS1 on its own.

MCS1 has the potential to impact thousands of students enrolled in FYE courses at Ohio State, as well as at other universities, by normalizing the assessment process. Further, this assessment can be incorporated into the curriculum as a benchmark which can then be used by faculty and administrators to make informed decisions about the curriculum based on student retention of key concepts.