Ascend HiFloat8 Format for Deep Learning

Yuanyong Luo, Zhongxing Zhang, Richard Wu, Hu Liu, Ying Jin, Kai Zheng, Minmin Wang, Zhanying He, Guipeng Hu, Luyao Chen, Tianchi Hu, Junsong Wang, Minqi Chen, Mikhaylov Dmitry, Korviakov Vladimir, Bobrin Maxim, Yuhao Hu, Guanfu Chen, Zeyi Huang·September 25, 2024

Summary

HiFloat8 (HiF8) is an 8-bit floating-point format for deep learning, offering tapered precision with 7 exponent values using 3-bit mantissa, 8 with 2-bit, and 16 with 1-bit. It extends denormal value encoding's dynamic range by 7 extra powers of 2, from 31 to 38 binades. HiF8 supports both AI training's forward and backward passes. The paper discusses HiF8's definition, rounding methods, and preliminary training and inference solutions. It showcases extensive simulation results on various neural networks, including traditional and large language models, to demonstrate HiF8's efficacy. HiF8 balances precision and dynamic range better than existing Float8 formats, using a flexible prefix code for the dot field, simplified exponent field, and mantissa field in biased form. HiF8 supports special values and is designed for both forward and backward passes in AI training.

Key findings

6
  • header
  • header
  • header

Tables

3

Advanced features