Continual Learning with Embedding Layer Surgery and Task-wise Beam Search using Whisper
Chin Yuen Kwok, Jia Qi Yip, Eng Siong Chng·January 14, 2025
Summary
The paper introduces Embedding Layer Surgery for Continual Learning in multilingual ASR, addressing Catastrophic Forgetting. It proposes separate token embedding copies for each language, selects one for transcription, and applies Task-wise Beam Search to correct errors. Compared to Experience Replay, the method reduces Average WER by 2.3% while maintaining performance on unseen languages.
Advanced features