Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation

FLAMe is a family of large language models designed for automatic evaluation tasks, demonstrating superior performance and lower bias compared to models like GPT-4. FLAMe variants, including FLAMe-24B, FLAMe-RM-24B, and FLAMe-Opt-RM-24B, significantly improve pass@1 accuracy on the HumanEval coding benchmark, surpassing other models. FLAMe is also effective in re-ranking decoded outputs, enhancing code generation performance by up to 40% for models like CodeGen-16B. FLAMe's zero-shot generalization abilities surpass models trained on proprietary data, and its variants outperform popular proprietary models across various autorater evaluation benchmarks. The text discusses advancements in evaluating large language models (LLMs) for better automatic evaluation, focusing on works like "TruthfulQA," "G-eval," and "Llms as narcissistic evaluators." It covers studies on instruction controllable summarization, data design for effective instruction tuning, learnable evaluation metrics for text simplification, zero-resource black-box hallucination detection, faithfulness and factuality in abstractive summarization, and the introduction of Meta Llama 3, the most capable openly available LLM. The text also highlights contributions such as "Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation," "Octopack: Instruction Tuning Code Large Language Models," "WebGPT: Browser-assisted Question-answering with Human Feedback," "CodeGen: An Open Large Language Model for Code with Multi-turn Program Synthesis," and updates on GPT models from OpenAI. The works collectively aim to improve the evaluation and understanding of LLM capabilities and limitations. In conclusion, the advancements in evaluating large language models, as discussed in the text, focus on improving the performance, generalization, and understanding of these models through various studies and contributions. The introduction of FLAMe and its variants, along with the works on instruction tuning, evaluation metrics, and detection of hallucinations, collectively aim to enhance the automatic evaluation of LLMs and provide a better understanding of their capabilities and limitations.