Agentless: Demystifying LLM-based Software Engineering Agents

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, Lingming Zhang·July 01, 2024

Summary

This collection of papers investigates the application of large language models (LLMs) in software engineering, particularly in the development of autonomous agents for code repair and assistance. AGENTLESS, a proposed agentless approach, simplifies the process into localization and repair, outperforming open-source agents in the SWE-bench Lite benchmark. It achieves high performance (27.33%) with low cost, demonstrating the potential of simple, interpretable techniques. Researchers address benchmark limitations by creating SWE-bench Lite-S for more rigorous evaluation. The studies highlight the benefits of agentless methods, question the need for complex agent-based systems, and emphasize the importance of evaluating models like GPT-4 and GPT-4o in tasks like code issue resolution and bug fixing. The papers also address challenges in benchmark descriptions, problem-solving granularity, and the role of AI in tasks such as code generation, translation, and fuzz testing. Overall, the research suggests that LLMs are increasingly contributing to software development efficiency and autonomy, while also calling for improvements in benchmark design and evaluation.

Key findings

6

Tables

3

Advanced features