AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations

This paper critically examines the use of Reinforcement Learning from Human Feedback (RLHF) and AI feedback (RLAIF) in aligning large language models with human values. It highlights the limitations of the 3H criteria (helpfulness, harmlessness, and honesty) in capturing the complexity of ethics and raises ethical concerns about user-friendliness, deception, flexibility, and system safety. The authors argue for a more nuanced and multidisciplinary approach, emphasizing the need to assess sociotechnical implications and consider alternative methods that integrate technical, philosophical, and ethical perspectives for a more comprehensive and robust AI alignment strategy. The papers collectively call for a shift away from simplistic technical solutions and towards a more inclusive, transparent, and accountable AI development process.