This article presents passive impedance control of flexible joint robots (FJRs) via inner-loop torque control of elastic joints. However, according to our theoretical analysis, the torque control methods of series elastic actuators (SEAs) are often limited by the fact that the acceleration signals are amplified by the control gains. Since the acceleration signals are often affected by differentiation noise, the analysis may become invalid in practice. To alleviate this limitation, we propose the use of the so-called series viscoelastic actuator (SvEA), which significantly reduces the acceleration amplification. Consequently, in contrast to the SEA case, the theoretical analysis of an SvEA-based FJR is valid in real implementations. We would like to highlight the fact that the theoretical analysis (more specifically, passivity analysis) is performed for nonlinear robot dynamics without linearization. As a result, the passive impedance controller can be realized more robustly with enhanced inner-loop torque control.