Construction robots are expected to frequently communicate in situ improvisations with human workers to adapt and change their workflow and methods. One way to achieve this is through deictic gestures that are one of the most effective forms of human–robot interaction (HRI) in delivering spatial information. Nevertheless, the limited coverage of deictic gestures in large-scale environments poses some challenges for both humans and robots in leveraging such techniques for HRI in construction. To identify the feasibility of deictic gestures in the construction domain and find applicable solutions for improving performance, this study aims to extend current knowledge on the performance in communicating positional information using deictic gestures by investigating the effects of spatial characteristics on spatial referencing, focusing on the target configuration, target distance, and relative position of human and robot. We observed that the recognition and estimation of deictic gestures were affected by the target plane, target position, and the target layout and that the robot performance was significantly reduced as the distance between the human and robot increased. The findings of this study demonstrate the challenges in spatial referencing within a large-scale environment and highlight the need for bidirectional communication in HRI.