How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles
A controlled logic-puzzle experiment found that lower-cost AI assistance induces more frequent AI use, and participants who request AI assistance during the AI-access phase perform worse at the task after assistance is removed. Greater independent problem-solving effort is associated with larger gains in latent ability.
The study examines the tension between short-term performance gains from AI assistance and long-term skill development. Participants completed logic puzzles before, during, and after AI availability, with varying AI request costs. Lower-cost assistance led to more frequent AI use. Those who used AI during the access phase performed worse after assistance was removed, and their unassisted performance was overestimated when predicted from earlier AI-assisted performance. A Bayesian latent ability model separated initial ability, post-AI ability, and skill change, showing that independent reasoning during AI access is linked to greater skill gains.
The research employs a Bayesian latent ability model to disentangle initial ability, post-AI ability, and participant-specific skill change. It estimates how independent reasoning during the AI-access phase relates to skill development, finding that greater independent problem-solving effort correlates with larger gains in latent ability. This suggests that AI assistance can substitute for independent reasoning, weakening skill development.
The findings imply that AI tools designed for on-demand assistance may inadvertently reduce user skill development if they encourage over-reliance. This has implications for educational technology, professional training, and AI product design, where balancing immediate performance support with long-term skill retention is critical.
For AI product developers and enterprises, the study highlights a potential trade-off: AI assistance can boost short-term productivity but may erode employee skills over time. This could affect training programs, performance evaluation, and the design of AI copilots to ensure they support rather than undermine human capability.
Future research may explore interventions to mitigate skill loss, such as prompting users to attempt problems independently before accessing AI, or designing AI assistance that scaffolds rather than replaces reasoning. The study also suggests the need for longitudinal studies to assess long-term skill trajectories in real-world settings.