Background: While psychological treatments are effective, a substantial portion of patients do not benefit enough. Early identification of those may allow for adaptive treatment strategies and improved outcomes. We aimed to evaluate the clinical usefulness of machine-learning (ML) models predicting outcomes in Internet-based Cognitive Behavioural Therapy, to compare ML-related methodological choices, and guide future use of these. Methods: Eighty main models were compared. Baseline variables, weekly symptoms, and treatment activity were used to predict treatment outcomes in a dataset of 6695 patients from regular care. Results: We show that the best models use handpicked predictors and impute missing data. No ML algorithm shows clear superiority. They have a mean balanced accuracy of 78.1% at treatment week four, closely matched by regression (77.8%). Conclusions: ML surpasses the benchmark for clinical usefulness (67%). Advanced and simple models perform equally, indicating a need for more data or smarter methodological designs to confirm advantages of ML.