Documentation Practices of Artificial Intelligence

Stefan Arnold, Dilara Yesilbas, Rene Gröbner, Dominik Riedelbauch, Maik Horn, Sven Weinzierl·June 26, 2024

Summary

The paper explores the evolution of AI documentation to ensure transparency and accountability in AI development and deployment. It highlights the progression from basic checklists like data and model cards to more comprehensive and automated approaches. Key findings include: 1. A diverse AI documentation landscape, encompassing data, model, and system cards, with a focus on addressing stakeholder needs and evolving ethical considerations. 2. A quantitative review of 39 studies reveals trends, challenges, and opportunities, with a shift towards more granular and interactive documentation. 3. Data cards, model cards, and system cards are prevalent, with model cards having the most dimensions and least automation, while data cards have the least. 4. Documentation practices are multimodal and primarily manual, with a need for broader coverage, particularly for end-users and regulatory compliance. 5. Researchers emphasize the importance of standardization, interactivity, and automation to improve AI documentation, including environmental, social, and compliance aspects. The paper concludes that AI documentation is evolving to be more holistic and inclusive, but automation remains underutilized. It calls for collaboration between academia and industry to refine practices and promote responsible AI development and governance.

Key findings

1

Introduction
Background
Historical context of AI documentation
Importance of transparency and accountability in AI
Objective
To analyze the development of AI documentation
Identify trends and challenges for stakeholder needs
Highlight the role of standardization and automation
Method
Data Collection
Literature review of 39 studies on AI documentation
Analysis of diverse documentation formats (data, model, system cards)
Data Preprocessing
Categorization and analysis of documentation features
Quantitative analysis of trends and automation levels
Data Cards
Prevalence and dimensions
Manual vs. automated processes
Focus on data quality and provenance
Model Cards
Dimensions and evolution
Automation levels and challenges
Importance for model explainability
System Cards
Overview and role in AI systems
Current state and manual efforts
Integration with other documentation types
Documentation Practices and Challenges
Multimodal and manual approaches
End-user and regulatory compliance coverage
Evolving ethical considerations
Standardization
Current standards and their impact
Need for harmonization and best practices
Interactivity and Automation
Advantages and limitations
Opportunities for future development
Conclusion
Holistic and inclusive documentation landscape
Underutilization of automation in AI documentation
Call for academia-industry collaboration
Recommendations for responsible AI development and governance
Basic info
papers
digital libraries
artificial intelligence
Advanced features
Insights
According to the quantitative review, which type of card (data, model, or system) has the most dimensions and the least automation, and why is this significant?
What are the main challenges and opportunities identified in the paper regarding the current state of AI documentation practices?
What recommendations does the paper make for improving AI documentation, particularly in terms of standardization and automation?
How has the evolution of AI documentation shifted from basic checklists to more comprehensive approaches, as mentioned in the paper?