KaPO: Knowledge-aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Models

The paper introduces KaPO, a Knowledge-aware Preference Optimization method for Retrieval-Augmented Language Models (RAG). KaPO addresses the issue of integrating external non-parametric evidence with internal parametric knowledge, which can lead to knowledge conflicts and confusion in model responses. The method aims to enhance controllable knowledge selection in real retrieval scenarios by learning to avoid negative signals through preference optimization. KaPO adjusts the balance between response length and preference data representation to improve adherence and noise robustness. Experiments show that KaPO outperforms previous methods in handling knowledge conflicts, with robust generalization across various out-of-distribution datasets, achieving over 37% improvement. KaPO is a Knowledge-aware Preference Optimization strategy designed to improve the reliability and interpretability of Large Language Models (LLMs) in generating responses. It tackles challenges such as simulating real-world retrieval scenarios, avoiding preference imbalance, and enhancing adherence and noise robustness. KaPO constructs comprehensive preference relations by simulating common error types and developing training strategies to avoid them. It also proposes a rewriting strategy to address length imbalance and a data ratio balancing strategy to address behavior pattern imbalance. KaPO has been validated on multiple models and datasets, showing improvements in performance and adaptability in out-of-distribution scenarios. The text discusses various methods to improve the reliability and interpretability of LLMs in generating responses. It highlights that modifying prompts does not significantly alter LLMs' internal prior token probabilities, potentially limiting the effectiveness of this approach. To address this, RAG incorporates external knowledge retrieval through prompt engineering, enhancing factual consistency and response quality. Intermediate contexts, such as commonsense, domain-specific knowledge, and chain-of-thought reasoning, are generated using pre-trained knowledge to improve final responses. However, these contexts may contain inaccuracies, leading to potential misdirection in retrieval or reader models. Knowledge editing is another method that focuses on updating model knowledge by identifying how models store factual knowledge and designing strategies to update pre-trained models. Challenges include unintentionally affecting unrelated parameters or causing inconsistencies within the model's internal knowledge. The text outlines a method for generating counterfactual answers to questions, aiming to create plausible but incorrect responses. It uses the SQuAD2.0 dataset, which contains human-annotated questions and answers across various domains. The process involves selecting relevant documents for questions with realistic answers and counterfactual answers. For questions with realistic answers, the corresponding document is chosen, while for counterfactual answers, the realistic answer is replaced in the document. Four documents, including one relevant and two on different topics, are shuffled to form the context. For scenarios without relevant documents, two hard and two easy irrelevant documents are selected, shuffled, and used as context. The text describes a method to fine-tune a Large Language Model (LLM) to address real-world challenges in Retrieval-Augmented Generation (RAG) tasks. The goal is to create a model, Θft, that can independently determine the context type and formulate responses, adhering to specific criteria. The criteria include providing a direct answer when the answer is present in the context (|S| = 1) and a general answer when the context is absent (|S| = 0). The text emphasizes the importance of the model's ability to avoid errors like contextual overinclusion and ignorance. KaPO is a Knowledge-aware Preference Optimization strategy designed to enhance large language models' adherence capability and noise robustness. It addresses two common error types in scenarios with varying contextual relevance: Contextual Ignorance and Contextual Overinclusion. KaPO utilizes instruction-tuning and negative gradient terms in the DPO comparative objectives to reduce the likelihood of unwanted responses. By aligning data lengths and balancing data ratios, it mitigates preference imbalances inherent in DPO. The strategy outperforms baseline methods across all metrics, showing significant improvements in RAd and RRo scores for the Squad2.0-Eval dataset. KaPO surpasses KAFT by using more complex contexts and comprehensive negative signals to enhance the models' capabilities. In conclusion, KaPO is a significant advancement in the field of large language models, addressing knowledge conflicts and improving adherence and noise robustness. It demonstrates superior performance in handling knowledge conflicts and outperforms previous methods in various out-of-distribution datasets. The method's ability to construct comprehensive preference relations, address length and behavior pattern imbalance, and fine-tune models for real-world scenarios showcases its potential for enhancing the reliability and interpretability of large language models.