Timely calls for help can really make a difference for elders who suffer from falls, particularly in private locations. Considering privacy protection and convenience for the users, in this paper, we approach the problem by using impulse-radio ultra-wideband (IR-UWB) monostatic radar and propose a learning model that combines convolutional layers and convolutional long short term memory (ConvLSTM) to extract robust spatiotemporal features for fall detection. The performance of the proposed scheme was evaluated in terms of accuracy, sensitivity, and specificity. The results show that the proposed method outperforms convolutional neural network (CNN)-based methods. Of the six activities we investigated, the proposed method can achieve a sensitivity of 95% and a specificity of 92.6% at a range of 8 meters. Further tests in a heavily furnished lounge environment showed that the model can detect falls with more than 90% sensitivity, even without re-training effort. The proposed method can detect falls without exposing the identity of the users. Thus, the proposed method is ideal for room-level fall detection in privacy-prioritized scenarios.