Improving Robustness of Compressed Models with Weight Sharing through Knowledge Distillation

Weight sharing is an effective technique in model compression for reducing size of deep learning models and enabling their deployment on resource-constrained devices. Existing weight sharing approaches mainly focus on maximizing compression ratios while maintaining model performance on test sets. In this study, we investigate weight sharing from a different perspective. This is in line with recently raised concerns about adversarial and non-adversarial robustness after compression, particularly in techniques such as pruning, which may adversely affect model performance in presence of adversarial and non-adversarial samples. As such, beyond the compression ratio, we explore the impact of weight sharing on a model’s robustness against corrupted data. Our performance evaluation on the CIFAR10-C and CIFAR100-C datasets, employing MobileNet and ResNet architectures, reveals that the impact of weight sharing methods on model performance is more significant in presence of corrupted data compared to the performance loss observed on normal data. Particularly, our assessment of ResNet18 with CIFAR100-C demonstrates that higher levels of compression ratios with weight sharing increase the models’ sensitivity to corruption. Furthermore, we show that retraining the model and iteratively applying weight sharing can enhance performance on both normal and corrupted data. To this end, we introduce a new knowledge distillation method that combines offline knowledge distillation and self-distillation, resulting in 1.6% improvement in robustness against corrupted data after weight sharing.