Authors:Xuanjun Chen, Haibin Wu, Helen Meng, Hung-Yi Lee, and Jyh-Shing Roger Jang
Abstract: Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications. However, to the best of our knowledge, the adversarial robustness of AVASD models hasn't been investigated, not to mention the effective defense against such attacks. In this paper, we are the first to reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks through extensive experiments. What's more, we also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples under an allocated attack budget. The loss aims at pushing the inter-class embeddings to be dispersed, namely non-speech and speech clusters, sufficiently disentangled, and pulling the intra-class embeddings as close as possible to keep them compact. Experimental results show the AVIL outperforms the adversarial training by 33.14 mAP (%) under multi-modal attacks.
The imperceptibility of Multi-Modality Attacks.
The word in red below is the answer.
1. Sample 1：a5mEmM6w_ks_1080_1140:22 (Speech)
1.1 Reference data (X)
1.2 Real data (A)
1.3 Adversarial data (B)
Please select the checkbox that you think is right in Sample 1.
(a) Reference data == Real data.
(b) Reference data == Adversarial data
2. Sample 2：JNb4nWexD0I_1440_1500:39 (Non speech)
2.1 Reference data (X)
2.2 Real data (A)
2.3 Adversarial data (B)
Please select the checkbox that you think is right in Sample 2.
(a) Reference data == Real data.
(b) Reference data == Adversarial data
3. Sample 3：JNb4nWexD0I_1740_1800:34 (Non speech)
3.1 Reference data (X)
3.2 Real data (A)
3.3 Adversarial data (B)
Please select the checkbox that you think is right in Sample 3.
(a) Reference data == Real data.
(b) Reference data == Adversarial data
4. Sample 4：rJKeqfTlAeY_1560_1620:46
4.1 Reference data (X)
4.2 Real data (A)
4.3 Adversarial data (B)
Please select the checkbox that you think is right in Sample 4.
(a) Reference data == Real data.
(b) Reference data == Adversarial data
5. Sample 5：5milLu-6bWI_1320_1380:52 (Speech)
5.1 Reference data (X)
5.2 Real data (A)
5.3 Adversarial data (B)
Please select the checkbox that you think is right in Sample 5.
(a) Reference data == Real data.
(b) Reference data == Adversarial data.
6. Sample 6：7YpF6DntOYw_1680_1740:66 (Non speech)
6.1 Reference data (X)
6.2 Real data (A)
6.3 Adversarial data (B)
Please select the checkbox that you think is right in Sample 6.
(a) Reference data == Real data.
(b) Reference data == Adversarial data
7. Sample 7：C25wkwAMB-w_0960_1020:18 (Speech)
7.1 Reference data (X)
7.2 Real data (A)
7.3 Adversarial data (B)
Please select the checkbox that you think is right in Sample 7.
(a) Reference data == Real data.
(b) Reference data == Adversarial data
8. Sample 8：qrkff49p4E4_1500_1560:3 (Non speech)
8.1 Reference data (X)
8.2 Real data (A)
8.3 Adversarial data (B)
Please select the checkbox that you think is right in Sample 8.
(a) Reference data == Real data.
(b) Reference data == Adversarial data.
9. Sample 9：1j20qq1JyX4_1020_1080:30 (Speech)
9.1 Reference data (X)
9.2 Real data (A)
9.3 Adversarial data (B)
Please select the checkbox that you think is right in Sample 9.
