For an erbium-doped fiber laser mode-locked by carbon nanotubes, we demonstrate experimentally and theoretically a new type of the vector rogue waves emerging as a result of the chaotic evolution of the trajectories between two orthogonal states of polarization on the Poincare sphere. In terms of fluctuation induced phenomena, by tuning polarization controller for the pump wave and in-cavity polarization controller, we are able to control the Kramers time, i.e. the residence time of the trajectory in vicinity of each orthogonal state of polarization, and so can cause the rare events satisfying rogue wave criteria and having the form of transitions from the state with the long residence time to the state with a short residence time.