Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers
A study (N = 809) across three deployment contexts (AI kidney allocation, AI agents simulating absent workers, and generative AI depictions of the deceased) examines the moral AI elicitation pipeline. It finds that developer choices in feature scoping, voter sampling, and question framing can shape the preferences produced. Morally relevant features shift across contexts, suggesting feature schemas should not be assumed to transfer across deployment domains.
As AI systems make more morally loaded decisions, moral preference elicitation polls participants on hypothetical dilemmas and uses aggregated votes to train policies. Before voting, developers make three key choices: feature scoping, voter sampling, and question framing. These choices are often opaque, undocumented, and treated as technical rather than normative. The study shows each choice can shape preferences produced by moral AI elicitation.
The research demonstrates that the moral AI elicitation pipeline is sensitive to developer decisions at three stages: feature scoping, voter sampling, and question framing. Morally relevant features shift across deployment contexts, indicating that feature schemas are not transferable across domains. This implies that elicitation pipelines require context-specific validation and documentation of normative choices.
Organizations deploying AI in morally loaded domains (healthcare allocation, worker simulation, digital resurrection) must recognize that preference elicitation is not a neutral technical process. Developer choices can systematically bias outcomes, creating legal, ethical, and reputational risks. Transparent documentation and stakeholder involvement in pipeline design are becoming necessary for responsible deployment.
For companies building morally consequential AI, understanding and controlling elicitation bias can reduce liability and improve public trust. Transparent elicitation processes may become a competitive differentiator in regulated industries. Conversely, ignoring these choices risks product failures, ethical backlash, or regulatory intervention.
Expect increased scrutiny of moral AI elicitation methods, with potential standards or guidelines for documenting feature scoping, sampling, and framing. Future research may focus on context-adaptive feature schemas and methods to audit or mitigate developer-induced bias. Regulatory interest in AI ethics may extend to the elicitation pipeline itself.