In exploring the societal readiness for shared autonomous vehicles (SAVs) in Swedish cities, this study profiles two distinct user groups—potential users and refusers—based on their willingness to share rides with strangers. By analyzing responses from a significant sample in Stockholm and Gothenburg, the study reveals key traits of these groups: potential users tend to be progressive, environmentally conscious men with public transport habits and positive experiences with AVs, while refusers are often women with traditional values, less formal education, and a preference for private cars, exhibiting concerns about safety and privacy. The research underscores the necessity of addressing the unique concerns of refusers to foster broader acceptance of SAVs. It highlights the potential of SAVs to revolutionize urban transport if societal concerns are aptly managed through policy and education, leveraging positive public transport experiences as a gateway to shared autonomous mobility.