Abstract: Shipborne surface wave radar is susceptible to the influence of complex sea conditions and platform motion, leading to a reduction in its detection capability for marine vessel targets. This issue is especially pronounced in dense multi-target areas and weak target areas. To address these challenges, this paper proposes an improved Mask Region Convolutional Neural Network (Mask R-CNN) target detection method for shipborne surface wave radar. Firstly, a Range-Doppler spectrum dataset of shipborne surface wave radar is constructed, and an adaptive feature extraction algorithm for vessel target samples is developed. Then, Mask R-CNN is introduced for target detection and instance segmentation of the Range-Doppler spectrum, combined with the Convolutional Block Attention Module to mine deep characteristic information of target echoes. Finally, the method is validated using measured data and the ship automatic identification system. Experimental results show that compared with the Mask R-CNN, YOLOv7, and background recognition CFAR methods, the proposed method not only reduces target detection leakage in conventional backgrounds but also effectively detects weak targets under strong sea clutter interference. Additionally, it identifies targets in dense areas where multi-vessel echoes overlap, thereby improving the detection capability of shipborne surface wave radar for marine vessel targets.