Realistic texture-based modelling methods, that is microstructural modelling and micromechanical modelling, are developed to simulate the rock aggregate breakage properties on the basis of the rock actual microstructure obtained using microscopic observations and image analysis. The breakage properties of three types of rocks, that is Avja, LEP and Vandle taken from three quarries in Sweden, in single aggregate breakage tests and in inter-aggregate breakage tests are then modelled using the proposed methods. The microstructural modelling directly integrates the microscopic observation, image analysis and numerical simulation together and provides a valuable tool to investigate the mechanical properties of rock aggregates on the basis of their microstructure properties. The micromechanical modelling takes the most important microstructure properties of rock aggregates into consideration and can model the major mechanical properties. Throughout this study, it is concluded that in general, the microstructure properties of rock aggregate work together to affect their mechanical properties, and it is difficult to correlate a single microstructure property with the mechanical properties of rock aggregates. In particular, for the three types of rock Avja, LEP and Vandle in this study, crack size distribution, grain size and grain perimeter (i.e. grain shape and spatial arrangement) show good correlations with the mechanical properties. The crack length and the grain size negatively affect the mechanical properties of Avja, LEP and Vandle, but the perimeter positively influences the mechanical properties. Besides, the modelled rock aggregate breakage properties in both single aggregate and inter-aggregate tests reveal that the aggregate microstructure, aggregate shape and loading conditions influence the breakage process of rock aggregate in service. For the rock aggregate with the same microstructure, the quadratic shape and good packing dramatically improve its mechanical properties. During services, the aggregate is easiest to be fragmented under point-to-point loading condition, and then in the sequence of multiple-point, point-to-plane and plane-to-plane loading conditions.