We propose a transceiver architecture for 5G fronthaul that adapts the modulation format according to the channel quality. The proposed solution operates in dual-mode using digital signal processing (DSP)-assisted analog radio-over-fiber (A-RoF) and digital radio-over-fiber (D-RoF) with multiple modulation orders. The system performance is assessed through Monte-Carlo simulations of the optical fiber link conveying wireless waveforms that utilize the filter bank multicarrier (FBMC) and orthogonal frequency-division multiplexing (OFDM) formats. The simulation results highlight the benefits of having both digital and DSP-assisted analog transmission capabilities to optimize the reach versus spectral efficiency trade-off in optical fronthaul.