Elderly persons’ perceived accessibility to railway traveling depends on their functional limitations/diseases, their functional abilities and their travel behaviors in interaction with the barriers encountered during whole trips. A survey was conducted on a random sample of 1000 city residents (65–85 years old; 57% response rate). The travels were perceived least accessible by respondents with severely reduced functional ability and by those with more than one functional limitation/disease (e.g., restricted mobility and chronic pain). Those who traveled “often”, perceived the accessibility to be better than those who traveled less frequently. For travelers with high functional ability, the main barriers to more frequent traveling were travel costs and low punctuality. For those with low functional ability, one’s own health was reported to be the main barrier. Our results clarify the links among existing functional limitations/functional abilities, the barriers encountered, the travel behavior, and the overall accessibility to traveling. By operationalizing the whole-trip concept as a chain of events, we deliver practical knowledge on vulnerable groups for decision-making to improve the transport environment for all.