AI literacy through the lenses of graduate students: insights from scenario analysis

Abstract

Generative artificial intelligence tools have gained notoriety since the launch of ChatGPT by OpenAI in November 2022. This technology is based on large language models, simulating a written ‘conversation’ between humans and algorithms, and is capable of generating text from the commands it receives. However, its use in academia is controversial and requires: knowledge of the tools (their strengths and weaknesses); the ability to evaluate sources and content generated by AI; interpersonal collaboration mediated by these systems; understanding the different contexts of use and the parameters for using AI in each context; autonomous and responsible interaction with AI; and consideration of the ethical implications of these tools (Allen & Kendeou, 2024). Considering the growing use of generative AI tools for learning and the complexity of the knowledge involved in their responsible use, the development of these skills – called artificial intelligence literacy – should be prioritised by teachers and students. The objective of this exploratory study was to analyse university students’ understanding of AI use in light of the theoretical framework of AI literacy developed by Allen and Kendeou (2024). The study employed a scenario analysis methodology to explore how graduate students assess the use of genAI in a hypothetical learning situation. Results pointed to difficulty in identifying ethics in the scenario, as shown by low ratings, and overlap with other components of the framework. The justifications unveiled students’ interpretation of ethics as behavioral trace instead of machine-related, as the proposed by the framework. Self-efficacy did not explain variance in scenario rating. These findings provide data on which AI literacy skills students clearly understand and which ones they may need instruction or assistance to develop, such as awareness of AI ethics. We discuss the relevance of theory and in particular the ED-AI lit framework to inform AI literacy practices in higher education.