UDHRBench: A Benchmark for LLM Human Rights Alignment Reveals Efficiency and Neutrality Rationales as Main Violation Drivers

01 ABSTRACT

Facts: The study released UDHRBench, containing 1,124 human rights questions covering all 30 articles of the UDHR and 4,496 responses from four models (ChatGPT, Grok, Gemini, Fanar), with expert annotations (κ=0.801); analysis found violations typically occur through seemingly rational reasoning such as appeals to efficiency, risk management, or scientific neutrality, rather than explicit rejection of rights; it also proposed an LLM-as-a-Judge framework achieving 89.8% weighted F1 and introduced an Alignment Leniency Score. Authors' view: There are systematic and non-trivial misalignments between models and international human rights standards; the automated framework can help detect human rights alignment issues at scale.

02 KEY FINDINGS

1. UDHRBench covers all 30 articles of the UDHR, with 1,124 human rights questions and 4,496 responses from four LLMs, and expert annotation agreement of κ=0.801.
2. Violations often appear through rationalized appeals to efficiency, risk management, or scientific neutrality rather than explicit rejection of rights.
3. The LLM-as-a-Judge framework detects human rights misalignment with 89.8% weighted F1 under zero-shot and few-shot prompting, and an Alignment Leniency Score quantifies judge models' resolution of ambiguous cases.