LLM-Based Open-Domain Integrated Task and Knowledge Assistants with Programmable Policies

KITA is a programmable framework for task-oriented conversational agents, offering reliable, grounded responses through controllable policies specified in a KITA Worksheet. Unlike Large Language Models (LLMs), KITA is resilient to diverse user queries, helpful with knowledge sources, and provides ease of programming policies through its declarative paradigm. It outperforms GPT-4 in execution accuracy, dialogue act accuracy, and goal completion rate, as demonstrated in a real-user study involving 62 participants. KITA enables explicit control and high-level support for integrated knowledge assistants, addressing limitations of traditional slot-filling frameworks and dialogue trees. The system uses a KITA Worksheet to manage dialogue flow, track user-supplied values, and support full compositionality between APIs and queries. It has been evaluated across three domains, showing superior performance compared to GPT-4 with function calling, particularly in restaurant reservations, ticket submissions, and course enrollments. The text discusses an AI system's interactions with users across various domains, focusing on issues with agent actions, error analysis, and user feedback. Key points include errors mainly occurring due to the assistant's failure to request specific details, leading to premature API calls and misinterpretation of user instructions. The AI demonstrates adaptability by recovering from errors through requesting additional information. Overall, user feedback is positive, praising the AI for accuracy, smooth conversation flow, and confirmation of selections. However, some users noted issues with response speed, restructuring queries, and a perceived forced conversation tone. A common issue is the AI's failure to seek confirmation before submitting final API calls, potentially leading to errors or incorrect actions. The text also describes a mapping between dialog acts in KITA and StarV2 banking systems, including 'AskField' for user information and 'Report' for status updates.