The presence of hazardous materials hinders the circular economy in construction and demolition waste management. However, traditional environmental investigations are costly and time-consuming, and thus lead to limited adoption. To deal with these challenges, the study investigated the possibility of employing registered records as input data to achieve in situ hazardous building materials management at a large scale. Through characterizing the eligible building groups in question, the risk of unexpected cost and delay due to acute abatement could be mitigated. Merging the national building registers and the environmental inventory from renovated and demolished buildings in the City of Gothenburg, a training dataset was created for data validation and statistical operations. Four types of inventories were evaluated to identify the building groups with adequate data size and data quality. The observations’ representativeness was described by plotting the distribution of building features between the Gothenburg dataset and the training dataset. Evaluating the missing data and the positive detection rates affirmed that reports and protocols could locate hazardous materials in the building stock. The asbestos and polychlorinated biphenyl (PCB)-containing materials with high positive detection rates were highlighted and discussed. Moreover, the potential inventory types and building groups for future machine learning prediction were delineated through the cross-validation matrix. The novel study contributes to the method development for assessing the risk of residual hazardous materials in buildings.