On the interface between Linguistics, Computer Science and Psychiatry: analyzing textual key-factors affecting BERT-based classification of schizophrenia in social media texts

Abstract

This paper investigates language impairments in schizophrenia (SZ) by integrating insights from language-centered investigations with computational approaches. Using BERT-base-cased, a transformer-based model, it explores how linguistic markers of SZ can be identified through Natural Language Processing (NLP) techniques, with emphasis on improving performance reliability via dataset refinement and approaching interpretability of deep learning outputs via statistical analyses of thematic content. We report the fine-tuning of a BERT model for text classification of 31,278 Reddit posts (15,639 SZ, 15,639 controls). The experiment evaluated the capacity of the model to distinguish language produced by individuals with SZ. The model achieved moderate performance (Accuracy = 0.6969; AUC = 0.78) and remained stable across hyperparameter configurations, indicating that foundation models such as BERT easily fit to data and, therefore, further performance gains are more likely to be derived from dataset refinement than from additional hyperparameter optimization. There were three key factors affecting the model’s performance: text length, topic of discussion and vocabulary choices. Posts that were correctly classified tended to be significantly longer (p < 0.001, M = 37.30), focused on specific abstract topics (e.g., religion), and contained more words related to mental conditions. These factors have also been reported in manual analyses of the impacts of SZ on language. These findings contribute to the accuracy of computational models aimed at working on linguistic classification tasks and underscore the value of carefully curated datasets, while demonstrating the viability of NLP methods in profiling SZ language.