Abstract: To address the challenges in mountainous highway rockfall detection, including scarce samples, variable target scales, and complex backgrounds that lead to weak model generalization and high miss rates, this paper proposes a YOLOv8-based rockfall object detection model integrating transfer learning and multi-strategy improvements. Firstly, a rockfall image dataset is constructed, covering different lithologies, multi-scale targets, and complex backgrounds, providing diverse support for model training. Secondly, to tackle the few-shot learning challenge, a transfer learning method based on ImageNet pre-training is introduced to avoid overfitting caused by training from scratch. Building on this, a progressive fine-tuning framework is established: the Coordinate Attention mechanism is embedded in shallow networks to suppress complex background interference; the Bidirectional Feature Pyramid Network replaces the original structure to enhance multi-scale feature fusion efficiency and improve sensitivity to small rockfalls; finally, the EIoU loss function is adopted to optimize bounding box regression, addressing inaccurate localization of irregular rockfalls. Experimental results show that compared to the baseline YOLOv8, the proposed model improves precision, recall, and mAP50 by 17.1%, 24.7%, and 17.4%, respectively, while maintaining low computational costs. It significantly reduces missed detections and false alarms of small targets in complex backgrounds. Moreover, the proposed model effectively enhances the detection accuracy and robustness of rockfall targets under few-shot conditions, providing a feasible technical solution for the development of intelligent rockfall monitoring and early warning systems on mountainous highways.