Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speeches
A paper proposes embedded conditional independence tests (eCITs) for testing whether LLM output carries information about an attribute beyond the source text. The method embeds text and applies existing CITs to representations. The paper includes a semi-synthetic simulation design to assess type I error control and power, and applies the method to German parliament speeches.
Researchers introduce embedded conditional independence tests (eCITs) to test conditional dependence between LLM-generated text and an attribute given a source text. By embedding the text and source, they transfer the null hypothesis to representation space under sufficiency conditions. They provide conditions for hypothesis equivalence and weaken sufficiency to mean sufficiency for conditional mean independence tests. A semi-synthetic simulation assesses type I error and power, with an application to German parliament speeches.
The eCIT framework leverages sufficient embeddings to transfer conditional independence testing from high-dimensional text to lower-dimensional representations. Sufficiency of the source embedding ensures validity of the embedded test for the original hypothesis. Mean sufficiency suffices when testing conditional mean independence. The semi-synthetic design enables controlled evaluation of type I error and power.
This method provides a statistical tool for auditing LLM outputs for leakage of sensitive attributes beyond the source text, relevant for compliance and fairness in generated content. Application to political speeches suggests use in detecting bias or unintended information in generated summaries or translations.
Enables verification that LLM outputs do not introduce spurious correlations or leak attributes, supporting trustworthy deployment in regulated domains such as government, legal, and healthcare text processing.
Potential next signals include empirical results on type I error and power from the simulation, extensions to other modalities, and adoption in LLM auditing pipelines. Further work may explore necessary conditions for sufficiency and practical embedding choices.