Unmanned Aerial Vehicles (UAVs) typically rely on Global Navigation Satellite Systems (GNSS) for localization. When GNSS signals are unreliable, UAV visual localization offers an effective alternative to determine the UAV’s position. This method matches and aligns UAV images with a satellite map to calculate the absolute localization result. However, the discrepancies between the UAV images and the satellite map present significant challenges to accurate matching and alignment, which can lead to localization inaccuracies or failures. To address these challenges, we propose RARFLoc, which fuses the relative localization process between UAV adjacent frames into the above absolute localization process to achieve precise localization even under challenging conditions. Notably, RARFLoc does not require prior knowledge of the initial position and incorporates mechanisms for failure detection and re-localization, greatly enhancing its robustness. Moreover, in the matching stage, we introduce a self-training boosted matching method that improves the matching model’s performance for remote sensing. During the alignment stage, a Depth-Guided Alignment (DGA) strategy is proposed to ensure accurate alignment between UAV images and the satellite map. Localization experiments were conducted to evaluate the performance of RARFLoc. The average localization errors of RARFLoc on the MSDI and Nanchang datasets are 3.73 m and 4.38 m, respectively, representing reductions of 40.4 % and 23.9 % compared with the baseline. The results show that it consistently delivers accurate and robust localization, even in environments with significant discrepancies between UAV images and the satellite map.