ChaosMining: A Benchmark to Evaluate Post-Hoc Local Attribution Methods in Low SNR Environments

This research evaluates post-hoc attribution methods in low signal-to-noise environments, focusing on their ability to distinguish predictive features from irrelevant ones across symbolic, image, and audio domains. The study develops synthetic datasets and benchmarks various models and attribution techniques (e.g., gradient-based saliency, DeepLift, Integrated Gradients, and Feature Ablation) under different noise conditions. Key findings include: 1. Gradient-based saliency is effective in feature selection but has scalability limitations. 2. Model generalization is linked to the efficacy of attribution methods. 3. A novel extension to Recursive Feature Elimination (RFE) for neural networks is introduced, addressing the lack of empirical evidence on their effectiveness. 4. Experiments on MLPs, CNNs, ViTs, and RNNs show that performance varies across modalities and noise types, with new metrics like Uniform Score and Functional Precision introduced. 5. Saliency Maps (SA) perform well in low SNR for symbolic data, while AlexNet and Vgg13 struggle due to architectural limitations. 6. RFEwNA, a modified RFE, improves feature selection for neural networks, especially in binary classification tasks. The research contributes to the understanding of XAI methods in diverse scenarios and highlights the need for future studies on more attribution techniques, diverse model configurations, and noise conditions.