Molecular dynamics (MD) methods are used to sample the time evolution of complex molecular systems, namely the transition events between configuration states. When these states are metastable, these transitions correspond to rare events and occur over very long-time scale, thus rendering the use of MD methods inefficient. The Parallel Replica algorithm, was designed to efficiently sample these rare events relying on a parallelization in time computation of the trajectory. Its mathematical formalization was carried out in the literature using the notion of quasi-stationary distribution (QSD), which can be seen as the long-time distribution of the dynamics trapped inside a state. Its existence was only proven recently for the underdamped Langevin dynamics (ULD) involved in the sampling of thermostated molecular systems. In this research letter we shall state the existence of a QSD for the ULD, for which we can provide a spectral interpretation. The existence of this QSD is then used to justify the application of Parallel Replica algorithm to the ULD. In addition, we shall extend known results regarding the behaviour of the ULD trajectories when the friction coefficient goes to infinity (overdamped limit). This allows us to provide an explicit expression of the overdamped limit of the previous QSD of the ULD. The findings of this study remain valid for ULD with non-conservative forces. They shall help for a better understanding of the mathematical framework underlying the Parallel Replica algorithm or related algorithms involving the QSD to the ULD. It also extends our knowledge on the overdamped limiting behaviour of the ULD.