Exploring the Stability Gap in Continual Learning: The Role of the Classification Head

Wojciech Łapacz, Daniel Marczak, Filip Szatkowski, Tomasz Trzciński·November 06, 2024

Summary

The text discusses the stability gap in continual learning, focusing on the impact of feature extractors and classification heads. It compares non-metric learning (NMC) with linear heads, showing NMC's superiority in stability and performance across various benchmarks. NMC improves final performance and stability, benefiting both randomly initialized and pre-trained networks. The research highlights that the stability gap is primarily caused by linear classification heads, not insufficient representations. NMC mitigates task-recency bias and is a simple, effective solution for enhancing stability in continual learning systems.

Key findings

3

Tables

2

Advanced features