Under seismic actions, stone masonry buildings are prone to damage. To assess the severity of damaged masonry buildings and their failure modes, engineers connect these problems to surface crack features, such as the crack width and the extent of cracking. We aim to further these assessments in this study, wherein we propose using simple machine learning models to predict: 1) three ratios encoding the degradation of stiffness, strength, and displacement capacity of damaged rubble stone masonry piers as a function of the observed crack features and the applied axial load and shear span ratio; and 2) the pre-peak vs. post-peak regime, based on the crack features. When predicting the stiffness, force, and drift ratios, the prediction error is significantly reduced when the axial load and shear span ratio are included in the feature vector. Furthermore, when predicting the pre-peak vs. post-peak regime, simple machine learning models such as the k-nearest neighbor and the logistic regression result in remarkable accuracy. The obtained results have significant implications on the automated post-earthquake assessment of masonry buildings using image data. It is shown based on documented laboratory test data, that, by selecting proper crack features and incorporating information about the kinematic and static boundary conditions, even simple machine learning models can predict accurately the damage level caused to a rubble masonry pier. The three crack features used in this study are the maximum crack width, length density, and complexity dimension. The pipeline developed in this paper is general enough and is applicable to other masonry typologies and elements upon new evaluation of crack features and image data.