360Zhinao Technical Report

The 360Zhinao Team has developed a 7B parameter language model, 360Zhinao-7B-Base, by addressing pretraining challenges through improved data cleaning, composition, and stable ablation experiments. They emphasize data balance and quality during alignment, extending context lengths with tailored data. The model uses RMs and RLHF for competitive performance, and the work highlights the importance of refining data strategies and context considerations in large language model development. Key contributions include insights on data processing, evaluation methods, and the release of open-source models. The study also explores data deduplication and its impact on model performance, with deduplication strategies shown to improve validation and specific task scores in evaluations like 360Eval and OpenCompass. The research contributes to the ongoing advancement of large language models and their applications.