MSTS: A Multimodal Safety Test Suite for Vision-Language Models

Abstract

Vision-language models are increasingly integrated into chat assistants and other consumer AI applications, but multimodal inputs can introduce safety risks that text-only evaluations miss. We introduce MSTS, a Multimodal Safety Test Suite containing 400 prompts across 40 fine-grained hazard categories. Each prompt combines text and an image whose unsafe meaning becomes clear only when interpreted together. Evaluations reveal safety problems in several open models, as well as models that appear safe because they fail to understand the prompts. We translate MSTS into ten languages, compare multimodal and text-only testing, and assess automated safety classifiers, finding that current classifiers remain inadequate.