A critical enabler of autonomous construction equipment is object detection, a computer vision task integral to navigation, task execution, and safety. However, challenging conditions at construction sites, such as mud splashes, dirt, and vibrations, can degrade object detection performance by causing sensor occlusions and image blurriness. Traditional adversarial training methods, which enhance model robustness by using perturbed data, are limited in construction environments due to the scarcity of diverse real-world adversarial data and the dynamic nature of these sites. Additionally, privacy concerns and site-specific data variability hinder data sharing across different construction sites. To overcome these challenges, this paper explores federated learning as a solution to enhance the robustness and adaptability of object detection models while preserving data privacy. FL enables continuous online learning without direct data exchange, offering a scalable and privacy-preserving approach to training models across diverse construction environments. Experimental results demonstrate that our approach improves model performance on the ConstScene dataset by up to ≈ 4.4% compared to the centralized AI model for object detection.