Learning Loss Landscapes in Preference Optimization

The study explores preference optimization algorithms' performance influenced by dataset properties, introducing a novel mirror descent-based framework that uses evolutionary strategies to discover loss functions for handling problematic scenarios. This approach leads to significant performance improvements over existing methods. The framework demonstrates generalization by applying the discovered loss functions to fine-tune large language models with mixed-quality data, outperforming traditional methods. The text also discusses the evaluation of offline preference optimization algorithms, focusing on ORPO, which outperforms DPO in MuJoCo environments with well-defined reward structures. Key findings include distinct failure modes of ORPO on low-quality datasets, relevant to practical applications. The study provides a proof for Theorem 3.1 in Appendix A, involving plugging equation (8) into a maximum likelihood problem (3) to derive the objective: π⋆ is the argmax of ED log σ. The text discusses the transferability of objectives learned using a specific methodology, showing that a static objective, discovered on the shuffled TLA dataset, successfully transfers to the shuffled Hopper dataset, achieving a final policy value of 1735±39. Additionally, when applied to the LLM-tuning task with shuffled data, the methodology outperforms the baseline, achieving 57% accuracy compared to 62% with the discovered objective. The trained model with the discovered objective also outperforms AlpacaE-val, achieving a 53% winrate, indicating its effectiveness across different domains.