Continual Learning with Embedding Layer Surgery and Task-wise Beam Search using Whisper

Chin Yuen Kwok, Jia Qi Yip, Eng Siong Chng·January 14, 2025

Summary

The paper introduces Embedding Layer Surgery for Continual Learning in multilingual ASR, addressing Catastrophic Forgetting. It proposes separate token embedding copies for each language, selects one for transcription, and applies Task-wise Beam Search to correct errors. Compared to Experience Replay, the method reduces Average WER by 2.3% while maintaining performance on unseen languages.

Key findings

4

Advanced features