EU countries have committed to taking national measures to reduce the average energy use of residential buildings by 16% by 2030 and 20–22% by 2035. Each country has decided how to achieve this and which buildings to focus on. However, certain categories of residential buildings, including historical buildings but also second homes, may be exempt from these obligations. In Sweden, second homes constitute a substantial part of the housing stock and contribute to energy use for heating and hot water of 2.79 TWh, of which 2.16 TWh is electricity. Even if the owners of second homes are not obliged to take energy efficiency measures, quite simple and inexpensive measures may contribute to substantial energy savings. In this study, simulations of energy use in 862 second homes heated with direct electric heating were carried out for five energy efficiency measures (additional attic insulation, new windows, air-to air heat pump and decreasing the temperature during absence to 10 and 5 °C). With the assumption that the 862 second homes are representative of the entire stock of electric-heated second homes in Sweden, the energy saving potential was estimated to be 1.17 TWh per year if all homes install heat pumps and 0.99 TWh and 1.30 TWh per year if all lower the temperature to 10 °C or 5 °C when the house is not used.