AI-Driven Classification of Personality Items: A Study on Large Language Models

Carlos Henrique Sancineto da Silva Nunes; Alexandre José de Souza Peres; Leonardo de Barros Mose; Petar Čolović; Ricardo Primi; Vithor Rosa Franco

doi:10.1590/1982-4327e3609

##article.authors##

Carlos Henrique Sancineto da Silva Nunes Universidade Federal de Santa Catarina https://orcid.org/0000-0002-7769-6937
- Conceptualization
- Data Curation
- Formal Analysis
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Resources
- Software
- Supervision
- Visualization
- Validation
- Writing – Original Draft Preparation
- Writing – Review & Editing
Alexandre José de Souza Peres Federal University of Mato Grosso do Sul https://orcid.org/0000-0002-3472-6120
- Conceptualization
- Formal Analysis
- Data Curation
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Resources
- Software
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing
Leonardo de Barros Mose University of San Francisco https://orcid.org/0000-0002-5328-7442
- Conceptualization
- Data Curation
- Formal Analysis
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Resources
- Software
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing
Petar Čolović University of Novi Sad https://orcid.org/0000-0003-1212-3131
- Data Curation
- Conceptualization
- Formal Analysis
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Software
- Resources
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing
Ricardo Primi University of San Francisco https://orcid.org/0000-0003-4227-6745
- Conceptualization
- Data Curation
- Formal Analysis
- Methodology
- Funding Acquisition
- Investigation
- Project Administration
- Resources
- Software
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing
Vithor Rosa Franco University of San Francisco https://orcid.org/0000-0002-8929-3238
- Conceptualization
- Data Curation
- Formal Analysis
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Resources
- Software
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing

DOI:

https://doi.org/10.1590/1982-4327e3609

Keywords:

personality, natural language processing, artificial intelligence, psychometrics, validity

Abstract

This study investigates the effectiveness of large language models (LLMs) in classifying items that evaluate the Big Five personality dimensions, focusing on performance variations across traits and comparing local and cloud-based models. Five Natural Language Processing (NLP) models were used to classify 385 personality items, employing three levels of detail in prompt design. The results indicate that larger generative models, such as ChatGPT-4o and Gemini 1.5 Pro, outperformed smaller models in terms of accuracy, both for the five personality factors and overall. However, the Llama 3.1 model, run locally, showed adequate results for judge-based analyses, offering a viable alternative for those prioritizing data privacy. The study highlights the potential of LLMs as complementary tools in the development of psychological assessment instruments.