In this work, we approach the problem of semantically segmenting fashion images into different categories of clothing. This problem poses particular challenges because of the importance of both textural information and cues from shapes and context. To this end, we propose a fully convolutional neural network based on feature pyramid networks (FPN), together with a backbone consisting of the ResNeXt architecture. Our experimental evaluation shows that the proposed model achieves state-of-the-art results on two standard fashion benchmark datasets, and a qualitative study verifies its effectiveness when applied to typical fashion images. The approach has a modest memory footprint and can be used without a conditional random field (CRF) without much degradation of quality which makes our model preferable from a computational perspective. When comparing all methods without a CRF, our approach outperforms all state-of-the-art models on both datasets by a clear margin in all evaluated metrics. In fact, our approach achieves a higher accuracy without the CRF than the state-of-the-art models using CRFs.