Human-In-the-Loop Software Development Agents

Wannita Takerngsaksiri, Jirat Pasuksmit, Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Ruixiong Zhang, Fan Jiang, Jing Li, Evan Cook, Kun Chen, Ming Wu·November 19, 2024

Summary

HULA, a human-in-the-loop LLM-based framework, was introduced for software development, allowing engineers to refine and guide Large Language Models in generating coding plans and source code. Evaluated through multi-stage assessments, HULA was deployed in Atlassian JIRA, showing benefits in minimizing development time and effort, especially for straightforward tasks. Challenges in code quality were identified. The framework integrates human-in-the-loop LLM-based agents for software development, emphasizing human intelligence and authority. It aims to collaboratively resolve tasks by dividing them into file localization, plan generation, and coding stages. The process involves setting up tasks, planning, coding, and reviewing code changes, with human oversight at each stage.

Key findings

8

Tables

1

Introduction
Background
Overview of HULA and its purpose in software development
Explanation of how Large Language Models (LLMs) are integrated into the framework
Objective
The goal of HULA in enhancing software development processes
Focus on minimizing development time and effort, particularly for simple tasks
Method
Data Collection
Techniques for gathering data for HULA's operation
Importance of data in training and refining LLMs within the framework
Data Preprocessing
Processes for preparing data to be used by HULA
Methods for ensuring data quality and relevance for LLMs
Deployment and Evaluation
Implementation in Atlassian JIRA
Description of HULA's integration into Atlassian JIRA
Benefits observed in the deployment phase
Multi-stage Assessments
Overview of the evaluation process for HULA
Stages of assessment and their outcomes
Challenges and Benefits
Code Quality
Identification of challenges in maintaining high code quality
Strategies for addressing code quality issues within HULA
Human Oversight
Role of human intervention in the development process
Importance of human intelligence and authority in HULA
Framework Components
Task Division
Explanation of how tasks are divided into file localization, plan generation, and coding stages
Collaboration between human and LLM agents
Process Stages
Detailed description of the process stages in HULA
Human oversight at each stage of task execution
Conclusion
Summary of HULA's impact
Recap of HULA's benefits and challenges
Future Directions
Potential improvements and future developments for HULA
Basic info
papers
human-computer interaction
software engineering
machine learning
artificial intelligence
Advanced features