Numerical Pruning for Efficient Autoregressive Models
Xuan Shen, Zhao Song, Yufa Zhou, Bo Chen, Jing Liu, Ruiyi Zhang, Ryan A. Rossi, Hao Tan, Tong Yu, Xiang Chen, Yufan Zhou, Tong Sun, Pu Zhao, Yanzhi Wang, Jiuxiang Gu·December 17, 2024
Summary
The paper introduces a training-free pruning method for decoder-only transformer-based autoregressive models, focusing on improving efficiency while maintaining performance for language and image generation tasks. It proposes a numerical score for Attention and MLP modules using Newton's method, and a compensation algorithm to enhance pruned model performance. The method achieves state-of-the-art results with reduced memory usage and faster generation speeds on GPUs, demonstrating effectiveness in compressing large autoregressive models for multiple generative tasks.
Introduction
Background
Overview of transformer-based autoregressive models
Challenges in deploying large models for language and image generation
Objective
Aim of the research: developing a training-free pruning method for efficiency improvement without compromising performance
Method
Numerical Score for Modules
Attention Module
Utilization of Newton's method for score calculation
Criteria for score determination
MLP Module
Similar score calculation approach
Different considerations for MLP modules
Compensation Algorithm
Objective: to mitigate performance degradation post-pruning
Mechanism of the compensation algorithm
Integration with the pruning process
Evaluation
Performance Metrics
Metrics used for assessing model efficiency and accuracy
Comparison with baseline models
Results on Multiple Tasks
Language generation tasks
Image generation tasks
Detailed performance metrics and improvements
Case Studies
GPU Utilization
Memory usage reduction
Generation speed enhancements
Real-world Applications
Examples of deploying pruned models in practical scenarios
Impact on resource-constrained environments
Conclusion
Summary of Contributions
Key findings and advancements
Future Work
Potential areas for further research
Recommendations for practical implementation
Basic info
papers
machine learning
artificial intelligence
Advanced features