Error vector magnitude (EVM) is commonly used for evaluating the quality of m-ary quadrature amplitude modulation (mQAM) signals. Recently proposed deep learning techniques for EVM estimation extend the functionality of conventional optical performance monitoring (OPM). In this article, we evaluate the tolerance of our developed EVM estimation scheme against various impairments in coherent optical systems. In particular, we analyze the signal quality monitoring capabilities in the presence of residual in-phase/quadrature (IQ) imbalance, fiber nonlinearity, and laser phase noise. We use feedforward neural networks (FFNNs) to extract the EVM information from amplitude histograms of 100 symbols per IQ cluster signal sequence captured before carrier phase recovery. We perform simulations of the considered impairments, along with an experimental investigation of the impact of laser phase noise. To investigate the tolerance of the EVM estimation scheme to each impairment type, we compare the accuracy for three training methods: 1) training without impairment, 2) training one model for all impairments, and 3) training an independent model for each impairment. Results indicate a good generalization of the proposed EVM estimation scheme, thus providing a valuable reference for developing next-generation intelligent OPM systems.