Eliciting Problem Specifications via Large Language Models

This paper explores the use of large language models (LLMs) in automating problem-solving by translating natural language descriptions into semi-formal specifications for cognitive systems. The authors design an LLM-enabled cognitive task analyst agent that generates problem-space definitions and strategies from AI literature. The goal is to reduce human intervention in problem formulation and enable AI systems to tackle diverse problems using domain-general methods. Key points include: 1. LLMs as a tool for automating problem representation, potentially speeding up AI research. 2. Problem spaces and cognitive task analysis (CTA) frameworks, like GOMS, are used to structure problem-solving processes. 3. The CTA agent generates problem specifications adhering to Polya's problem-solving principles and Newell's definition of problem spaces. 4. The agent aims to create problem formulations for weak methods, allowing for autonomous problem-solving. 5. Two alternative system designs are discussed, one relying heavily on LLMs and another with more human intervention. The study highlights the potential of LLMs in streamlining AI development and suggests future directions for multi-modal inputs and more sophisticated problem-solving capabilities. However, it also acknowledges the need for refining problem representations and addressing limitations in current models.