In recent years, the significance of applying Unmanned Surface Vehicles (USVs) in coastal defense has been progressively increasing. Precise prediction of USV trajectories plays a vital role in the decision-making of coastal defense, anti-privacy, and so on. However, the intricate nature of USV trajectories, characterized by high maneuverability and sudden motion pattern changes, poses great challenges for accurate prediction. To address these issues, this paper proposes a trajectory prediction model based on an improved Transformer with sparse self-attention and physical rule constraints. Focus on designing the “Max-Mean” sparse self-attention mechanism to streamline computational demands and memory usage, and the physical loss function to improve the accuracy and robustness of predictions. Moreover, a generative decoder is included to improve the model’s ability to process long sequence data and the inference efficiency. To verify the prediction effect of the proposed method, we construct a USV simulation trajectory dataset based on the ship kinematic model for trajectory prediction experiments. The simulation results illustrate that the proposed model surpasses existing trajectory prediction models and fulfills the stringent requirements for precise and rapid USV trajectory predictions.