CausalDS: Data Science Agents Move from Code Generation to Structural Causal Inference and Refusal to Answer
The CausalDS preprint, submitted on July 9, 2026, proposes a causal evaluation for data science agents, covering structural causal models, Pearl's three-level causal hierarchy, and refusal to answer when evidence is insufficient.
Current data agents excel at generating queries, charts, and correlation analyses, but business decisions require distinguishing observation, intervention, and counterfactuals. This study incorporates causal structure and refusal-to-answer capabilities into evaluation, making the gap between 'being able to produce analysis' and 'being able to support action decisions' visible.
The benchmark constructs association, intervention, and counterfactual questions around structural causal models, and checks whether the model can abstain when unidentifiable or information is insufficient. It simultaneously tests reasoning correctness and the model's ability to recognize information boundaries; final answer format or code runnability only covers part of the capability.
Enterprise data copilots that cannot identify causal assumptions and evidence boundaries may package correlations as strategic recommendations; acceptance of data agents needs to add causal hierarchy, assumption disclosure, and refusal rate.
Before using data agents for pricing, growth, or operational decisions, they should be required to clarify causal graphs, identify assumptions and unanswerable conditions, and have domain experts review intervention recommendations, rather than only accepting SQL and charts.
This study is still a preprint and a controlled benchmark. It needs to be validated in dirty data, hidden confounders, time series, and real organizational metrics, and compare the differences between tool-augmented agents and pure models.