Vision Language Models (VLMs) have shown remarkable capabilities in multimodal under-standing, yet their susceptibility to adversarial perturbations poses a significant threat to their reliability in real-world applications. Despite often being imperceptible to humans, these perturbations can drastically alter model outputs, leading to erroneous interpretations and decisions. This paper introduces DiffCAP, a novel diffusion-based purification strategy that can effectively neutralize adversarial corruptions in VLMs. We theoretically establish a provable recovery region in the forward diffusion process and meanwhile quantify the convergence rate of semantic variation with respect to VLMs. These findings manifest that adversarial effects monotonically fade as diffusion unfolds. Guided by this principle, DiffCAP leverages noise injection with a similarity threshold of VLM embeddings as an adaptive criterion, before reverse diffusion restores a clean and reliable representation for VLM inference. Through extensive experiments across six datasets with three VLMs under varying attack strengths in three task scenarios, we show that DiffCAP outperforms existing defense techniques by a substantial margin. Notably, DiffCAP significantly reduces both hyperparameter tuning complexity and the required diffusion time, thereby accelerating the denoising process. Equipped with theorems and empirical support, DiffCAP provides a robust and practical solution for securely deploying VLMs in adversarial environments. The source code is available at https://github.com/JasonFu1998/DiffCAP

N-linked glycosylation (N-glyc) sites (N-X-S/T, X≠P) can be introduced by somatic hypermutation in immunoglobulin Fab regions. In patients with rheumatoid arthritis (RA), anti-citrullinated protein antibodies have a striking overrepresentation of Fab N-glycosylation. To further explore this, we sequenced B cell receptors (BCRs) from peripheral blood of 13 RA patients and 6 healthy control subjects, analyzing in total >250,000 heavy chain (VH) and >100,000 light chain sequences from both total B cells and citrullinated fibrinogen-reactive (Cit-Fib+) cells. Distribution of variable VH genes in VDJ DNA, and transcripts of unmutated IgM and class-switched BCR, revealed transcript gene-usage bias and higher VH4 in natural rearrangements by out-of-frame VDJ DNA in RA patients compared with control subjects. IgG Fab N-glyc sites were slightly more prominent in RA than control subjects (14.9% versus 12.1%; P = 0.048) with certain VH genes (e.g. VH1-18, VH1-69, VH3-9) displaying enriched N-glyc cumulative frequencies by somatic hypermutation. VH gene N-glyc hotspots were identified, explained by a lower threshold for codon conversion (i.e. K/S/T-X-S/T) and especially frequent in VH4s. Yet, RA patients had significantly more N-glyc in complementarity-determining region (CDR) 1 and 3 compared with control subjects. Expanded clonotypes with somatic hypermutation-induced N-glyc sites were delineated by network analysis in both the RA patients and control group, but patients with RA displayed more highly mutated class-switched members. Furthermore, Cit-Fib+ BCR-expanded clonotypes could be traced in the total B cell repertoire and exhibited increased frequency of N-glyc in mutated IgG/IgA subsets. Our findings highlight how Fab N-glyc sites can be linked to biased clonotype evolution and B cell selection in chronic responses

In the ever-evolving adversarial machine learning landscape, developing effective defenses against patch attacks has become a critical challenge, necessitating reliable solutions to safeguard real-world AI systems. Although diffusion models have shown remarkable capacity in image synthesis and have been recently utilized to counter lp-norm bounded attacks, their potential in mitigating localized patch attacks remains largely underexplored. In this work, we propose DiffPAD, a novel framework that harnesses the power of diffusion models for adversarial patch decontamination. DiffPAD first performs super-resolution restoration on downsampled input images, then adopts binarization, dynamic thresholding scheme and sliding window for effective localization of adversarial patches. Such a design is inspired by the theoretically derived correlation between patch size and diffusion restoration error that is generalized across diverse patch attack scenarios. Finally, DiffPAD applies inpainting techniques to the original input images with the estimated patch region being masked. By integrating closed-form solutions for super-resolution restoration and image inpainting into the conditional reverse sampling process of a pre-trained diffusion model, DiffPAD obviates the need for text guidance or fine-tuning. Through comprehensive experiments, we demonstrate that DiffPAD not only achieves state-of-the-art adversarial robustness against patch attacks but also excels in recovering naturalistic images without patch remnants. The source code is available at https://github.com/JasonFu1998/DiffPAD. 

We introduce a new task called Referring Atomic Video Action Recognition (RAVAR), aimed at identifying atomic actions of a particular person based on a textual description and the video data of this person. This task differs from traditional action recognition and localization, where predictions are delivered for all present individuals. In contrast, we focus on recognizing the correct atomic action of a specific individual, guided by text. To explore this task, we present the RefAVA dataset, containing 36, 630 instances with manually annotated textual descriptions of the individuals. To establish a strong initial benchmark, we implement and validate baselines from various domains, e.g., atomic action localization, video question answering, and text-video retrieval. Since these existing methods underperform on RAVAR, we introduce RefAtomNet – a novel cross-stream attention-driven method specialized for the unique challenges of RAVAR: the need to interpret a textual referring expression for the targeted individual, utilize this reference to guide the spatial localization and harvest the prediction of the atomic actions for the referring person. The key ingredients are: (1) a multi-stream architecture that connects video, text, and a new location-semantic stream, and (2) cross-stream agent attention fusion and agent token fusion which amplify the most relevant information across these streams and consistently surpasses standard attention-based fusion on RAVAR. Extensive experiments demonstrate the effectiveness of RefAtomNet and its building blocks for recognizing the action of the described individual. The dataset and code will be made publicly available at RAVAR.

Dance improvisation is an active research topic in the arts. Motion analysis of improvised dance can be challenging due to its unique dynamics. Data-driven dance motion analysis, including recognition and generation, is often limited to skeletal data. However, data of other modalities, such as audio, can be recorded and benefit downstream tasks. This paper explores the application and performance of multimodal fusion methods for human motion recognition in the context of dance improvisation. We propose an attention-based model, component attention network (CANet), for multimodal fusion on three levels: 1) feature fusion with CANet, 2) model fusion with CANet and graph convolutional network (GCN), and 3) late fusion with a voting strategy. We conduct thorough experiments to analyze the impact of each modality in different fusion methods and distinguish critical temporal or component features. We show that our proposed model outperforms the two baseline methods, demonstrating its potential for analyzing improvisation in dance