How Generative AI Can Radically Improve Multiple Choice Tests
The multiple choice test has been a mainstay of science education for decades, even though most teachers recognize it to be stale and flawed. Now, two scientists who focus on improving biology and chemistry education have put forth an idea to update the multiple choice test with generative AI, the type of artificial intelligence behind ChatGPT.
Mike Klymkowsky, a Professor in Molecular Cellular & Developmental Biology at the University of Colorado-Boulder, and Melanie M. Cooper, a Professor of Chemisry focused on Chemistry Education at Michigan State University, recently published their proposal to the arXiv preprint database. Thus, it has not yet been peer-reviewed.
The duo developed a chatbot trained to evaluate student responses based on various sources related to cell and molecular biology. They dubbed it "Dewey," after the education reformer John Dewey. He believed that students thrive in an environment where they are allowed to experience and interact with the curriculum, rather than just having it shown to them.
"Dewey, the bot, is able to analyze student answers to a wide range of questions, providing a detailed summary of necessary facts and concepts that are missing or misapplied in student answers, as well as generating suggestions for instructional approaches to address them," they explained.
With Dewey, Klymkowsky and Cooper envision a key change to multiple choice tests. Rather than students simply selecting the answer they think is correct, students also explain why they chose their answer and why each other option is incorrect.
"Dewey is then asked to analyze those responses, and then supply the instructor with that analysis," the researchers wrote. "Response analysis and the generation of instructional suggestions by Dewey is essentially immediate. The result is that the instructor has the time needed to thoughtfully consider how to adapt their instructional efforts, to introduce or review background materials, and to engage students with targeted in-class questions."
In a "proof of concept" study, Klymkowsky and Cooper offered two multiple choice questions to a small group of graduate students, then gave their answers to Dewey to score and analyze. Dewey then offered specific instructional suggestions so the teacher could subsequently clear up any prominent misconceptions after the brief exam.
For example, in one suggestion, Dewey recommended using "examples and counterexamples to teach what is meant by 'directed' in scientific contexts, clarifying that it refers to physical influences like gradients rather than purposeful actions."
With AI bots like Dewey, Klymkowsky and Cooper think it's possible for testing in science to focus on students' understanding of concepts rather than their ability to memorize information.
"Previously, in most cases the analysis and reporting of the results of open-ended formative assessment activities has been impractical or impossible to deliver in a timely and/or economical manner. Through the use of AI-based analyst bots, like Dewey, it becomes possible to quickly generate in-depth analyses at both the overall class level and the individual student level and deliver them to instructors. Such class level analyses of multiple choice/ concept inventory-type questions and other formative assessments provides a richer and more readily applied evidence base about student understanding than the percent of correct answers."
In an email interview, Klymkowsky told RCS that AI and Google have made bulk memorization essentially obsolete, as scientific facts and information can be readily retrieved. AIs like Dewey grant STEM educators a key opportunity to make their courses more about understanding and accurately apply underlying concepts and principles, which would produce better scientists, doctors, engineers, citizens, etc.
"It is clear that AI-based bots could transform educational outcomes dramatically. The 24/7 availability of knowledgeable tutor bots might also impact inclusion, retention, and success," he said.
Source: Michael Klymkowsky & Melanie M. Cooper. The end of multiple choice tests: using AI to enhance assessment. arXiv:2406.07481
