A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs
Myeongsoo Kim, Tyler Stennett, Saurabh Sinha, Alessandro Orso·November 11, 2024
Summary
AutoRestTest is a multi-agent framework for black-box REST API testing, integrating MARL, SPDG, and LLMs. It optimizes testing by treating it as a separable problem, with four agents focusing on APIs, dependencies, parameters, and values. SPDG simplifies dependency search, while LLMs handle value restrictions. Evaluated on 12 real-world services, AutoRestTest outperforms leading tools in code, operation coverage, and fault detection, identifying internal server errors where others fail.
Introduction
Background
Overview of black-box REST API testing
Importance of automated testing in API development
Challenges in black-box testing of complex APIs
Objective
To present AutoRestTest as a novel solution for black-box REST API testing
Highlighting its integration of MARL, SPDG, and LLMs
Emphasizing its effectiveness in optimizing testing processes
Method
Data Collection
Techniques for gathering data on APIs, dependencies, and parameters
Importance of comprehensive data for effective testing
Data Preprocessing
Methods for cleaning and organizing data to enhance testing efficiency
Role of preprocessing in improving the performance of MARL, SPDG, and LLMs
Multi-Agent Framework
Agents and Their Roles
Overview of the four agents in AutoRestTest
Responsibilities of each agent in the testing process
Integration of MARL, SPDG, and LLMs
Explanation of how these technologies work together to optimize testing
Specific contributions of each technology to the overall framework
Evaluation
Test Environment
Description of the real-world services used for evaluation
Criteria for selecting these services
Performance Metrics
Code coverage
Operation coverage
Fault detection capabilities
Results
Comparison of AutoRestTest against leading tools
Key findings on performance improvements
Case Studies
Detailed Analysis
Examination of specific scenarios where AutoRestTest outperformed other tools
Highlighting the identification of internal server errors
Impact on Testing Processes
Discussion on how AutoRestTest enhances the efficiency and effectiveness of testing
Potential for broader application in API testing
Conclusion
Summary of Contributions
Recap of AutoRestTest's innovative approach to black-box REST API testing
Future Directions
Potential areas for further research and development
Considerations for integrating AutoRestTest into existing testing frameworks
Basic info
papers
software engineering
artificial intelligence
Advanced features
Insights
How does AutoRestTest perform in comparison to leading tools in terms of code, operation coverage, and fault detection?
What is AutoRestTest and how does it work for black-box REST API testing?
Which components are integrated into AutoRestTest for optimizing testing processes?
What specific techniques does AutoRestTest use to identify internal server errors effectively?