Breast cancer is a heterogeneous disease where the tumor microenvironment, including extracellular components, plays a crucial role in tumor progression, potentially modulating treatment response. Different approaches have been used to develop three-dimensional models able to recapitulate the complexity of the extracellular matrix. Here, we use cell-free patient-derived scaffolds (PDSs) generated from breast cancer samples that were recellularized with cancer cell lines as an in vivo-like culture system for drug testing. We show that PDS cultured MCF7 cancer cells increased their resistance against the front-line chemotherapy drugs 5-fluorouracil, doxorubicin and paclitaxel in comparison to traditional two-dimensional cell cultures. The gene expression of the environmentally adapted cancer cells was modulated in different ways depending on the drug and the concentration used. High doses of doxorubicin reduced cancer stem cell features, whereas 5-fluorouracil increased stemness and decreased the proliferative phenotype. By using PDSs repopulated with other breast cancer cell lines, T-47D and MDA-MB-231, we observed both general and cell line specific drug responses. In summary, PDSs can be used to examine the extracellular matrix influence on cancer drug responses and for testing novel compounds in in vivo-like microenvironments.