Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level

The paper introduces iLR-DPO, an iterative method that enhances large language models by incorporating a length penalty to reduce verbosity while improving alignment with human preferences. Starting with a 7B parameter model, iLR-DPO achieves a 50.5% length-controlled win rate against GPT-4 Preview on AlpacaEval 2.0, outperforming competitors across various benchmarks. The research contributes an open-source model, focusing on minimizing alignment tax and enhancing language comprehension. The study also explores the use of reward models, beam search, and best-of-n sampling, with ablation studies highlighting the importance of length control and iterative training. While the model demonstrates improved alignment, it relies on GPT-4 as a reference and has limitations in reducing verbosity without significantly increasing response length. The work provides a framework for future research on model optimization and alignment.