The recent breakthroughs in machine learning (ML) and deep learning (DL) have catalyzed the design and development of various intelligent systems over wide application domains. While most existing machine learning models require large memory and computing power, efforts have been made to deploy some models on resource-constrained devices as well. A majority of the early application systems focused on exploiting the inference capabilities of ML and DL models, where data captured from different mobile and embedded sensing components are processed through these models for application goals such as classification and segmentation. More recently, the concept of exploiting the mobile and embedded computing resources for ML/DL model training has gained attention, as such capabilities allow the training of models via local data without the need to share data over wireless links, thus enabling privacy-preserving computation by design, model personalization and environment adaptation, and deployment of accurate models in remote and hardly accessible locations without stable internet connectivity. This work summarizes and analyzes state-of-the-art systems research that allows such on-device model training capabilities and provides a survey of on-device training from a systems perspective.