Long-haul high speed optical transmission systems are significantly distorted by the interplay between the electronic chromatic dispersion (CD) equalization and the local oscillator (LO) laser phase noise, which leads to an effect of equalization enhanced phase noise (EEPN). The EEPN degrades the performance of optical communication systems severely with the increment of fiber dispersion, LO laser linewidth, symbol rate, and modulation format. In this paper, we present an analytical model for evaluating the performance of bit-error-rate (BER) versus signal-to-noise ratio (SNR) in the n-level phase shift keying (n-PSK) coherent transmission system employing differential carrier phase estimation (CPE), where the influence of EEPN is considered. Theoretical results based on this model have been investigated for the differential quadrature phase shift keying (DQPSK), the differential 8-PSK (D8PSK), and the differential 16-PSK (D16PSK) coherent transmission systems. The influence of EEPN on the BER performance in term of the fiber dispersion, the LO phase noise, the symbol rate, and the modulation format are analyzed in detail. The BER behaviors based on this analytical model achieve a good agreement with previously reported BER floors influenced by EEPN. Further simulations have also been carried out in the differential CPE considering EEPN. The results indicate that this analytical model can give an accurate prediction for the DQPSK system, and a leading-order approximation for the D8PSK and the D16PSK systems.