AgentFAIR · Jul 17, 2026
AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets
AgentFAIR is a multi-agent framework for evaluating FAIR compliance of geospatial datasets. It uses structured metadata extraction and 13 sub-principle-specific LLM evaluators, each producing a 0-3 maturity score, cited evidence, and recommendations. A critic agent checks evidence and consistency, and can request re-evaluation. On 50 datasets from 10 repositories, mean Findability, Accessibility, Interoperability, and Reusability scores were 79.7%, 70.4%, 45.3%, and 72.0%. Rank correlations with four baseline tools ranged from 0.31 to 0.61; the FAIR-enough comparison was not statistically significant. On a 10-dataset repeated-run subset, sub-principle agreement averaged 89%.
What happened
AgentFAIR is a multi-agent framework designed to evaluate the FAIRness of geospatial datasets. It combines structured metadata extraction with 13 LLM-based evaluators, each assessing a specific FAIR sub-principle on a 0-3 maturity scale, providing cited evidence and recommendations. A critic agent ensures consistency and can request re-evaluation. In tests on 50 datasets from 10 repositories, the framework achieved mean scores of 79.7% (Findability), 70.4% (Accessibility), 45.3% (Interoperability), and 72.0% (Reusability). Rank correlations with existing tools were moderate (0.31–0.61), and sub-principle agreement on repeated runs averaged 89%.
Technical significance
The framework employs a modular multi-agent architecture where specialized LLM evaluators handle individual FAIR sub-principles, each generating structured outputs with evidence citations. A critic agent performs consistency checks and triggers re-evaluation, improving reliability. The approach addresses limitations of existing tools, such as handling JavaScript-rendered pages and repository-specific identifiers, by leveraging LLM capabilities for flexible evidence gathering.
Industry impact
Current FAIR evaluation tools produce inconsistent results, with standard deviations of normalized scores up to 30.3 percentage points across tools for the same dataset. AgentFAIR demonstrates that LLM-based multi-agent systems can provide more consistent and explainable assessments, potentially setting a new standard for automated data quality evaluation in scientific data management.
What to watch
Further validation on larger and more diverse datasets is needed to establish statistical significance against existing tools. Integration with data repositories and adoption by data stewards could streamline FAIR compliance workflows. The approach may be extended to other domains requiring structured evaluation of digital objects.
Decision value
Improved FAIRness evaluation can enhance data discoverability and reuse, benefiting sectors reliant on geospatial data such as urban planning, climate modeling, and disaster response. Automated, consistent assessments reduce manual effort and support compliance with funding and publishing mandates.