This paper presents a cost-benefit analysis of the Multi-Terminal Voltage Source Converter based HVDC (VSC-MTDC) using a proposed Mixed ac/dc Optimal Power Flow (M-OPF). In this analysis, the operational benefits of VSC-MTDC systems are evaluated and compared with their corresponding investment costs to derive the benefit-to-cost ratios (BCR) for selected candidates of VSC-MTDC systems in the transmission expansion planning. Case studies using a modified Nordic 32-bus system have been carried out. The results have shown that the embedded VSC-MTDC system might bring out a reduction in total operation cost depending on its configuration and location. The proposed M-OPF model incorporates a detailed model of a VSC-MTDC, and is suitable to the operation and planning analysis of the ac transmission grid with the embedded VSC-MTDC system.