Unequal Verdicts: Investigating Gender Bias in LLM-Based Fake News Detection
A study published on arXiv on 2026-08-04 systematically investigates gender bias in LLM-based fake news detection using the LIAR benchmark augmented with gender variants of speaker job titles. Six state-of-the-art LLMs were evaluated. All models exhibited gender sensitivity: 9.79%–35.13% of statements received inconsistent labels across neutral, male, and female variants, with male-female flip rates of 6.5%–23.6%. Five models showed statistically significant directional effects, with the strongest displaying male-skeptic patterns.
Large language models used for automated fact-checking show significant gender bias, as demonstrated by a new study that augmented the LIAR dataset with gender-varied job titles. Across six LLMs, veracity judgments changed for up to 35% of statements depending solely on the perceived gender of the speaker. The findings reveal both instability and systematic favoritism, with male-skeptic bias being the most pronounced directional effect. This highlights a critical reliability and fairness issue in AI-driven misinformation detection.
The study introduces a controlled methodology by augmenting the LIAR benchmark with three gender variants (Neutral, Male, Female) of speaker job titles, isolating gender as the sole variable. It measures bias through inconsistency rates and directional effect statistics, revealing that LLMs' fact-checking decisions are not robust to gender cues. The male-skeptic pattern suggests that models may have learned associations between male speakers and lower credibility from training data.
As LLMs are increasingly integrated into content moderation and fact-checking pipelines, this research underscores a pressing need for bias audits before deployment. The observed flip rates of up to 23.6% between male and female variants could lead to disproportionate censorship or amplification of certain voices, posing reputational and regulatory risks for platforms and news organizations.
For companies deploying LLM-based fact-checking, these findings indicate a risk of biased outcomes that could undermine trust and lead to legal challenges. Investing in bias detection and mitigation can differentiate products in the trust and safety market. Conversely, failure to address such biases may result in user backlash and increased scrutiny from policymakers.
Next signals include the development of bias mitigation techniques specifically for fact-checking tasks, such as counterfactual data augmentation or fairness constraints during fine-tuning. Expect follow-up studies on intersectional biases (e.g., race and gender) and real-world impact assessments. Regulatory bodies may begin to require fairness evaluations for automated content moderation systems.