Workers’ motion trajectories or spatial tracks on construction sites contain useful safety-related information. Existing safety management research on worker trajectory typically analyzes the interactions between worker motion trajectory and risk sources. The historical accident-free zone of a group of workers is a reflection of the potential safety zones on a construction site. Few studies have investigated construction workers’ trajectory safety risks from the integrated perspectives of group and hazard sources. Therefore this study developed a novel and integrated safety risk diagnosis method combining hazard source and group movement distributions to fully utilize the phone Global Positioning System (GPS) trajectory information of construction workers. The proposed method diagnoses workers’ risk exposures by considering workers’ trajectories in unsafe and safe areas. In addition, the method uses expert confidence and comprehensive decision indexes to determine the feature weights. Furthermore, the proposed safety risk diagnosis method adopts the grey optimization Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) diagnosis model to diagnose workers’ risk management priorities and behavior adjustment direction. An actual project site was used to assess the performance of the proposed diagnostic method. The results show that the developed method provides a quantitative means for project managers to measure workers’ spatial-temporal risk exposure, diagnose safety risks, and plan for safety controls. The proposed integrated method provides a new perspective to make full use of workers’ trajectory information and helps to provide practical and specific data-driven safety guidance for construction managers and workers.