Using Computer Vision to Model Fashion Outfit Compatibility

Abstract

Fashion retailers generate large amounts of clothing related data such asclothing item images, clothing item metadata, and outfits. Computer vi-sion can aid in the task of building outfits by creating a model that learnsbothsimilaritybetween clothing items that are interchangeable andcom-patibilitybetween clothing items of different type that go well togetherin an outfit. To achieve this, a model needs to compare images acrossvarious similarity conditions such as color, shape, and category. A recentstate-of-the-art method named Similarity Condition Embedding Network(SCE-Net) learns multiple similarity conditions without explicit supervi-sion from a unified embedding space that produce image embeddings thatcan be used to score outfits. In this paper, we examine the performance ofthis network on outfit compatibility and fill-in-the-blank tasks for an on-line clothing retail dataset from H&M to better understand how the net-work learns concepts of similarity and compatibility in the fashion do-main. To further explore its performance we also create a messaging appthat acts as a virtual stylist by using the trained model.