Language models and political bias: do LLMs reflect public opinion in Brazil?

Abstract

This article examines whether large language models (LLMs) reflect or diverge from the political preferences of the populations that use them. Existing research typically measures political bias using abstract ideological scales or benchmarks derived from Anglo-American contexts, implicitly assuming cross-national comparability. We propose an alternative approach that conceptualizes bias in terms of comparison with the public opinion of a given population. Using the Brazilian Electoral Study (ESEB) 2022 as a benchmark, we compare responses from four widely used LLMs — ChatGPT, DeepSeek, Gemini, and Grok — across 22 political questions covering five thematic domains. Each model was queried 50 times per question, allowing estimation of both central tendencies and response variability. Results indicate systematic but multidirectional misalignment between LLM outputs and Brazilian public opinion. In the questions about democracy, models tend to be more protective of democratic institutions than the average Brazilian. Models also tend to adopt more liberal positions on moral and diversity issues, while Brazilians take more punitive stances on criminal justice than the models; and, counterintuitively, models show, in average, weaker support for some redistributive programs than the electorate. We also observed differences between models, although in lesser degree. The findings contribute methodologically by introducing population-specific benchmarks for evaluating political bias and substantively by extending the debate beyond Anglo-American contexts, raising implications for the role of AI systems as political information intermediaries in diverse democracies.