Structural health monitoring (SHM) and Non-destructive Damage Identification (NDI) using responses of structures under dynamic excitation have an imperative role in the engineering application to make the structures safe. Interpretations of structural responses known as inverse problems are emerging topics with a large body of works in the literature. They have been widely solved with Machine Learning (ML) techniques such as Artificial Neural Network (ANN), Deep Neural Network (DNN), Adaptive Network-based Fuzzy Inference System (ANFIS), and Support Vector Machine (SVM). Nonetheless, these approaches can precisely predict the inverse problems of civil structures (e.g., truss or frame systems) with low damage levels, which have to wait until the structures reach certain damage or deteriorate level. The issue is related to the fact that most of the real structures have very low damage levels during their routine maintenances and usually be neglected due to limitations of the current techniques. This paper proposes a combination of Particle Swarm Optimization and Support Vector Machine (PSO-SVM) for damage identifications. The proposed approach is inspired by the effective searching capability of PSO, which can eliminate the redundant input parameters and robust SVM technique to classify damage locations effectively. In other words, natural frequencies and mode shapes extracted from the numerical examples of truss and frame structures are used as input parameters in which the redundant parameters might lead to reduction of the accuracy in the predicting models. The proposed PSO-SVM shows superior accuracy prediction in both damage locations and damage levels compared to the other ML models. It also substantially outperforms other ML models through validated cases of low damage levels.