Prompt Optimization with Human Feedback

This paper investigates the problem of optimizing prompts for large language models (LLMs) without numeric performance scores, focusing on scenarios where human feedback is the primary evaluation method. The authors propose Automated Prompt Optimization with Human Feedback (APOHF), a dueling bandits-inspired algorithm that efficiently selects prompt pairs for user preferences. APOHF is applied to tasks such as improving user instructions, text-to-image models, and refining responses, showing its effectiveness in finding high-performing prompts with minimal feedback. The study demonstrates APOHF's superiority over baselines in tasks like optimizing user instructions and response generation, even with limited data. The algorithm's practicality is highlighted, as it addresses real-world challenges where direct performance evaluation is not feasible, and the potential for misuse is acknowledged, calling for future research on security measures.