Language of images: Classifying marketing images with transformers and vision language models

Abstract

Visual communication is central to marketing. With the help of convolutional neural networks (CNNs) marketing has labeled large image datasets to understand visual impact. However, CNNs focus on local cues (e.g., smiles). They can miss marketing-relevant meanings shaped by context and configuration (e.g., joyful vs. sarcastic smiles). Recent advances like transformer-based vision models (TVMs) apply text-analytical concepts to image data. Vision language models (VLMs) such as GPT-5 or Phi-4 jointly represent images and text. These richer linguistic representations might succeed in classifications where CNNs fall short. Unlike CNNs, pretrained VLMs require no additional training, even for new image-related tasks. However, it remains unclear how accurate they classify marketing-relevant labels. Which of these paradigms and classification models should marketing rely on? Is any single model best suited for all applications? Drawing on prior marketing publications, we identify 18 datasets covering what and who appears in images, and how images are perceived. VLMs such as GPT-5 and Phi-4 achieve state-of-the-art accuracy across a wide range of image-related tasks without requiring task-specific fine-tuning. However, they should not be trusted blindly. They can result in unexpectedly high error rates for some tasks. A multi-paradigm ensemble of TVMs and VLMs can overcome these challenges. We conclude with recommendations when to test which models.