Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study

Membership inference attacks (MIAs) seek to infer whether a data point was used in training. We develop practical MIAs for large-scale multi-modal models like CLIP, overcoming computational challenges via (i) cosine-similarity thresholding between text and image features with augmentation aggregation, and (ii) a weakly supervised attack using ground-truth non-members. CLIP models are shown susceptible: simple baselines exceed 75% accuracy, and enhanced attacks improve average-case performance by 17% and are ≥7× more effective at low FPRs—highlighting privacy risks in foundational multi-modal models.