Inertial measurement units (IMUs) are increasingly used to estimate movement quality and quantity to the infer the nature of motor behavior. The current literature contains several attempts to estimate movement smoothness using data from IMUs, many of which assume that the translational and rotational kinematics measured by IMUs can be directly used with the smoothness measures spectral arc length (SPARC) and log dimensionless jerk (LDLJ-V). However, there has been no investigation of the validity of these approaches. In this paper, we systematically evaluate the use of these measures on the kinematics measured by IMUs. We show that: (a) SPARC and LDLJ-V are valid measures of smoothness only when used with velocity; (b) SPARC and LDLJ-V applied on translational velocity reconstructed from IMU is highly error prone due to drift caused by integration of reconstruction errors; (c) SPARC can be applied directly on rotational velocities measured by a gyroscope, but LDLJ-V can be error prone. For discrete translational movements, we propose a modified version of the LDLJ-V measure, which can be applied to acceleration data (LDLJ-A). We evaluate the performance of these measures using simulated and experimental data. We demonstrate that the accuracy of LDLJ-A depends on the time profile of IMU orientation reconstruction error. Finally, we provide recommendations for how to appropriately apply these measures in practice under different scenarios, and highlight various factors to be aware of when performing smoothness analysis using IMU data.