To address the issues of significant multi-scale variations in target and insufficient accuracy in complex scenarios during the highway potholes detection in aerial images by UAV, an improved YOLOv11 algorithm for aerial highway potholes detection is proposed. Firstly, a lightweight enhanced detection module (LEDM_module) is designed in the backbone network to replace the original C3K2 feature extraction module. By using the grouped parallel processing method, the multi-scale pothole feature channels are segmented, and the key information of potholes is dynamically extracted in the form of adaptive feature enhancement. The combination of lightweight computing and redundant parameter elimination is utilized to improve the extraction accuracy of pothole features and the operational efficiency of the model. Secondly, the enhanced multi-scale attention fusion module (EMSA_module) is used in the neck network to replace the original feature fusion method,which based on upsampling concatenation and convolution. By integrating dynamic attention calibration of pothole feature weights, grouped spatial refinement of pothole edges, and residual feature fusion, the cross-scale information transmission efficiency is enhanced under the conditions of diluted small pothole features and confusion between large pothole features and background. The experimental results show that the mAP@50 and mAP@0.5-0.95 of the improved model are 86.6% and 58.3%, respectively, compared with the baseline YOLOv11n, the above indicators have improved by 5.74% and 11.69%, respectively. The recall rate is 82.7%, compared with YOLOv11n, the indicator has improved by 19.68%. The experimental results prove that the optimization strategy proposed in this paper can effectively improve the detection ability for multi-scale and weak features potholes, and reduce the missed detection rate for the task of highway potholes detection in aerial images by UAV.