AI-Driven Classification of Personality Items: A Study on Large Language Models

Carlos Henrique Sancineto da Silva Nunes; Alexandre José de Souza Peres; Leonardo de Barros Mose; Petar Čolović; Ricardo Primi; Vithor Rosa Franco

doi:10.1590/1982-4327e3609

article.authors6a050d3ea26ea

Carlos Henrique Sancineto da Silva Nunes Universidad Federal de Santa Catarina https://orcid.org/0000-0002-7769-6937
- Conceptualization
- Data Curation
- Formal Analysis
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Resources
- Software
- Supervision
- Visualization
- Validation
- Writing – Original Draft Preparation
- Writing – Review & Editing
Alexandre José de Souza Peres Universidade Federal de Mato Grosso do Sul https://orcid.org/0000-0002-3472-6120
- Conceptualization
- Formal Analysis
- Data Curation
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Resources
- Software
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing
Leonardo de Barros Mose Universidade São Francisco https://orcid.org/0000-0002-5328-7442
- Conceptualization
- Data Curation
- Formal Analysis
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Resources
- Software
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing
Petar Čolović University of Novi Sad https://orcid.org/0000-0003-1212-3131
- Data Curation
- Conceptualization
- Formal Analysis
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Software
- Resources
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing
Ricardo Primi Universidade São Francisco https://orcid.org/0000-0003-4227-6745
- Conceptualization
- Data Curation
- Formal Analysis
- Methodology
- Funding Acquisition
- Investigation
- Project Administration
- Resources
- Software
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing
Vithor Rosa Franco Universidade São Francisco https://orcid.org/0000-0002-8929-3238
- Conceptualization
- Data Curation
- Formal Analysis
- Funding Acquisition
- Investigation
- Methodology
- Project Administration
- Resources
- Software
- Supervision
- Validation
- Visualization
- Writing – Original Draft Preparation
- Writing – Review & Editing

DOI:

https://doi.org/10.1590/1982-4327e3609

Keywords:

personality, natural language processing, artificial intelligence, psychometrics, validity

Resumen

This study investigates the effectiveness of large language models (LLMs) in classifying items that evaluate the Big Five personality dimensions, focusing on performance variations across traits and comparing local and cloud-based models. Five Natural Language Processing (NLP) models were used to classify 385 personality items, employing three levels of detail in prompt design. The results indicate that larger generative models, such as ChatGPT-4o and Gemini 1.5 Pro, outperformed smaller models in terms of accuracy, both for the five personality factors and overall. However, the Llama 3.1 model, run locally, showed adequate results for judge-based analyses, offering a viable alternative for those prioritizing data privacy. The study highlights the potential of LLMs as complementary tools in the development of psychological assessment instruments.