This article addresses the dysfunction of protection systems in the presence of FACTS devices, with a focus on resolving the issues of overreach and underreach commonly encountered by conventional distance relays. The article proposes the implementation of an adaptive distance relay that exploits a novel technique to tackle these challenges. The relay utilizes a modified boundary characteristic in the form of a quadrilateral zone. The boundaries of the rectified zone are determined using a 5th-degree polynomial equation. To strike a balance between improving adaptive boundary accuracy and minimizing zone expansion, the coefficients of the selected boundaries and their intersections are obtained through a consolidated application of a multi-objective genetic algorithm, strengthened by utilizing the Pareto front technique. Through extensive studies conducted on the enhanced quadrilateral zone of the distance relay, the effectiveness of this approach in establishing a robust protection system is demonstrated. By employing this adaptive distance relay approach, the occurrence of dysfunction in distance relays operation in the presence of FACTS devices can be significantly reduced compared to conventional relays.