Abstract: Shipborne surface wave radar is susceptible to the influence of complex sea conditions and platform motion, which leads to the reduction of its detection capability for marine vessel targets, especially the detection of dense multi-target area and weak target area is more difficult. Aiming at the above problems, this paper proposes an improved Mask Region Convolutional Neural Network (Mask R-CNN) target detection method for shipborne surface wave radar. Firstly, the Range-Doppler spectrum spectral dataset of shipborne surface wave radar is constructed, and the adaptive feature extraction algorithm of shipborne surface wave radar vessel target samples is developed; then Mask R-CNN is introduced into the target detection and instance segmentation of the Range-Doppler spectrum of shipborne surface wave radar, which is combined with the Convolutional Block Attention Module to realize the mining of target echo deep characteristic information; Finally, the method of this paper is validated by the measured data and the ship automatic identification system. The experimental results show that compared with the Mask R-CNN, YOLOv7 and background recognition CFAR methods, the method proposed in this paper not only reduces the target leakage detection under the conventional background, but also effectively detects the weak targets under the interference of strong sea clutter as well as the targets in the gathering area where the multi-vessel echoes overlap with each other, which improves the detection capability of the shipborne surface-wave radar for the targets of vessels at sea.