Semantic segmentation with image-level annotation has been widely studied for its friendly annotation and satisfactory performance. Aiming at the problem of sparse activation regions and semantic ambiguity between foreground and background of class activation maps, a dual-branch weakly supervised semantic segmentation network based on activation modulation is proposed. The network uses Resnet50 and Vision Transformer as a two-branch feature extraction network, and designs an activation modulation module embedded in the convolutional branch, which forces the model to activate the intermediate fraction of pixels to generate a compact class activation map, thus alleviating the problem of sparse activation regions of class activation maps. Second, a dynamic threshold adjustment strategy based on cosine annealing decay is proposed, which adaptively determines the highest background threshold during the training process, so that more low-confidence foreground pixels are involved in the segmentation training, and complete and accurate segmentation maps are generated. The effectiveness of the network is verified on the PASCAL VOC 2012 as well as MS COCO 2014 datasets. mIou values are 74.2% and 74.0% on the PASCAL VOC 2012 validation and test sets, respectively, and 45.9% on the MS COCO 2014 validation set. The experimental results show that the network can solve the mis-segmentation problem and achieve excellent segmentation performance in scenes with similar front background colours.