The death of the short-form physics essay in the coming AI revolution
Will Yeadon, O. Inyang, Arin Mizouri, Alex Peach, Craig P. Testrow
Abstract
Abstract The latest AI language modules can produce original, high quality full short-form (300-word) Physics essays within seconds. These technologies such as ChatGPT and davinci-003 are freely available to anyone with an internet connection. In this work, we present evidence of AI generated short-form essays achieving First-Class grades on an essay writing assessment from an accredited, current university Physics module. The assessment requires students answer five open-ended questions with a short, 300-word essay each. Fifty AI answers were generated to create ten submissions that were independently marked by five separate markers. The AI generated submissions achieved an average mark of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mn>71</mml:mn> <mml:mo>±</mml:mo> <mml:mn>2</mml:mn> <mml:mi mathvariant="normal">%</mml:mi> </mml:math> , in strong agreement with the current module average of <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mn>71</mml:mn> <mml:mo>±</mml:mo> <mml:mn>5</mml:mn> <mml:mi mathvariant="normal">%</mml:mi> </mml:math> . A typical AI submission would therefore most-likely be awarded a First Class, the highest classification available at UK universities. Plagiarism detection software returned a plagiarism score between <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mn>2</mml:mn> <mml:mo>±</mml:mo> <mml:mn>1</mml:mn> </mml:math> % (Grammarly) and <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mn>7</mml:mn> <mml:mo>±</mml:mo> <mml:mn>2</mml:mn> </mml:math> % (TurnitIn). We argue that these results indicate that current natural language processing AI represent a significant threat to the fidelity of short-form essays as an assessment method in Physics courses.