In recent years, modulation recognition, as a key technology in wireless communication signal processing, has faced challenges such as insufficient open-set recognition capability and limited utilization of input correlation in deep learning models in complex open environments. To address these issues, this paper proposes a modulation signal open-set recognition method that integrates complex-valued attention and multi-dimensional loss functions. Specifically, this method introduces a multi-dimensional attention mechanism into the deep separable complex-valued network structure, effectively mining the correlation features between signal amplitude and phase, and achieves multi-loss fusion optimization through a decoder-assisted feature extraction, including smooth label cross-entropy, dynamic center constraint, and reconstruction error, to enhance the feature distribution discrimination and model generalization ability. Experiments on the public dataset RadioML2016.10a show that this method achieves a classification accuracy of 95% for known categories in the closed-set recognition task, and in the open-set recognition scenario, the recognition accuracy for known categories is 93%, the detection rate for unknown categories is 86%, and the overall open-set recognition performance is 89%. These results demonstrate excellent adaptability to open environments.