The construction site environment is quite complex with many dangerous hazards (e.g., foundation pits, holes). To avoid injuries, workers must wear helmets that are color-coded for the specific type of work, which is helpful to identify whether workers are in permitted areas. Therefore, it is possible to identify unauthorized intrusion by classifying the safety helmets. This study proposes a vision-based method called Helmet–Yolov5 to automatically detect unauthorized intrusions by workers on construction sites. Multiple improvement measures are made to enhance the model performance. First, the attention mechanism is used to enhance the weights of object regions in the image, which makes the detection of small objects more effective. Second, atrous spatial pyramid pooling is adopted to preserve the detail information of the image. Third, the universal upsampling operator is introduced to fuse image features at different scales. To verify the effectiveness of the improved model, images collected from a real construction site are used to build a large-scale image dataset of safety helmets for model testing. It shows that the proposed Helmet-Yolov5 model is more accurate than the original Yolov5 model, also with high inference speed. Compared to other state-of-the-art models (e.g., Yolov4), the Helmet-Yolov5 model has considerable advantages in term of high detection accuracy and efficiency.