Unmanned Aerial Vehicles (UAVs) have immense potential to enhance Intelligent Transport Systems (ITS) by aiding in real-time traffic monitoring, emergency response, and infrastructure inspection, leading to rich data collection, lower response times, and efficient urban mobility management. Machine learning (ML) is a crucial component in UAV-assisted ITS as it processes UAV-captured data in both the perception layer and decision layers of intelligent components for vehicle/pedestrian detection, trajectory optimization, and resource allocation. Importantly, the integration of UAVs and cutting-edge deep learning (DL) techniques is fostering an exciting synergy, equipping UAVs with unparalleled intelligence and autonomy, particularly, for the perception layer of UAVs. Despite these enhancements, their usefulness for detection and traffic extraction tasks remains largely unexplored. The contributions of this paper are divided into two main aspects: (1) UAVs in different ITS application scenarios that are empowered by ML technologies are reviewed. (2) A thorough survey aiming to explore a quantitative understanding of widely used DL models via a series of experiments and comparisons is presented. Four DL models, namely Convolution Neural Network (CNN), regions with CNN (R-CNN), Faster R-CNN, and You Only Look Once (YOLO)), in combination with different backbones, are designed and employed on five aerial datasets. Finally, we present a discussion of the remaining challenges and future works.