投审稿入口
Local-global collaborative multi-scale feature augmentation for hyperspectral and multispectral image fusion
-
摘要:
针对单一传感器成像受物理条件与系统设计限制，难以同时兼顾高空间分辨率与高光谱分辨率的问题，提出一种局部−全局协同的多尺度特征增强融合方法，用于高光谱图像与多光谱图像融合任务，旨在充分挖掘2种模态在空间结构与光谱信息上的互补优势，在保持光谱一致性的同时显著提升空间细节表达能力。该方法的整体框架由特征提取、特征融合、特征增强和图像重建4个模块协同构成。特征提取模块分别对高光谱与多光谱图像进行多层次编码，获得初始的光谱与空间特征表示；特征融合模块在统一特征空间内对多源特征进行交互与对齐；在此基础上，构建局部−全局特征增强模块，其中局部增强子模块通过多种卷积块与多尺度感受野强化纹理、边缘等细粒度空间细节，全局增强子模块引入光谱−空间融合的Transformer结构并结合多尺度卷积操作，以建模长程依赖关系并提升全局上下文与光谱一致性表达能力；最后，图像重建模块将增强后的融合特征映射回图像空间，生成高质量融合结果。在多个公开数据集上的定量与定性试验结果表明，所提方法在空间细节保持、光谱保真度以及多项综合评价指标（如：PSNR（峰值信噪比）、SSIM（结构相似度）、SAM（光谱角相似度）、ERGAS（相对全局自适应误差））方面均优于现有主流融合方法，融合图像的空间分辨率显著提升，光谱失真小，视觉效果与客观指标均表现优异，且在不同场景下均展现出良好的鲁棒性。本研究提出的局部−全局协同多尺度特征增强方法有效缓解了单一传感器成像中空间与光谱分辨率相互制约的问题，能够生成兼具高空间细节与高光谱保真度的融合图像，为遥感图像融合提供了一种高性能、强鲁棒的解决方案，具有良好的应用潜力。
-
关键词:
- 高光谱−多光谱融合 /
- 多尺度特征增强 /
- 局部−全局特征 /
- 跨模态融合 /
- Transformer
Abstract:ObjectiveImages acquired by a single remote sensing sensor are inherently constrained by hardware and physical limitations, making it difficult to simultaneously achieve high spatial resolution and high spectral resolution. Hyperspectral images provide rich spectral information but typically suffer from low spatial resolution, whereas multispectral images contain finer spatial details at the cost of reduced spectral fidelity. To address this trade-off, this study proposes a local-global collaborative multi-scale feature augmentation method for hyperspectral and multispectral image fusion. The objective is to fully exploit the complementary spatial and spectral characteristics of heterogeneous data sources, thereby generating fused images that preserve spectral consistency while significantly enhancing spatial detail expression.
MethodsThe proposed fusion framework consisted of four cooperative modules: Feature extraction, feature fusion, feature augmentation, and image reconstruction. First, the feature extraction module independently encoded the hyperspectral and multispectral inputs using dedicated convolutional layers to obtain hierarchical spectral and spatial feature representations. Second, the feature fusion module integrated the extracted features into a shared latent space, enabling cross-modal interaction and alignment. The core component was the feature augmentation module, which was divided into local and global sub-modules. The local feature augmentation sub-module employed multiple convolutional blocks with different receptive fields to strengthen fine-grained spatial details such as edges, textures, and local structures. The global feature augmentation sub-module introduced a spectral-spatial fusion Transformer architecture combined with multi-scale convolutions to model long-range dependencies and enhance global contextual information as well as spectral consistency. Finally, the image reconstruction module mapped the augmented fusion features back to the image domain to produce the final high-resolution hyperspectral image.
ResultsExtensive experiments were conducted on several benchmark hyperspectral and multispectral datasets, including both quantitative evaluations and qualitative visual comparisons. The proposed method consistently outperformed state-of-the-art fusion methods across multiple evaluation indicators. In terms of spatial detail preservation, the fused images exhibited sharper edges and clearer textures with significantly improved spatial resolution. Regarding spectral fidelity, the proposed method achieved low spectral distortion, with SAM (spectral angle mapper) values comparable to those of the best-performing competitors. Comprehensive evaluation indicators such as PSNR, SSIM, and ERGAS also demonstrated superior performance. For example, on the widely used CAVE and Harvard datasets, the proposed method achieved average PSNR improvements of 1.5-2.5 dB over the best baseline methods. Visual comparisons further confirmed that the proposed method effectively avoided common artifacts such as blurring and spectral aliasing. Moreover, the method showed robust performance across different scenes and varying degradation conditions.
ConclusionThe proposed local-global collaborative multi-scale feature augmentation method effectively mitigates the inherent spatial-spectral trade-off in single-sensor imaging systems. By jointly enhancing local fine-grained details and global contextual dependencies, the method generates fused images with both high spatial resolution and high spectral fidelity. Experimental results demonstrate its superiority over existing approaches in terms of accuracy, robustness, and visual quality. The proposed framework provides a powerful and versatile solution for hyperspectral and multispectral image fusion, with strong potential for practical applications in remote sensing, environmental monitoring, and beyond.
-
图 1 总体框架
Hr-MSI. 高分辨率多光谱图像；Lr-HSI. 低分辨率高光谱图像；Hr-HSI. 高分辨率高光谱图像；下同
Figure 1. Overview of architecture
图 2 光谱−空间融合Transformer子模块
Y1. 空间特征图；Z1. 光谱特征图；X1. 最终的光谱−空间融合特征；RH×W×C. 图像的高度，宽度和通道数；下同
Figure 2. Spectral-spatial fusion transformer block
图 3 局部特征增强模块（Concat. 在最后1个维度上的拼接；X2. 局部特征增强后得到的特征图）
Figure 3. Local augmented convolution block
图 5 Washington DC Mall数据集的融合结果（R：54；G：34；B：10）
（R：54；G：34；B：10）为用于合成假彩色图像的合成波段；a1～j1为在 RGB（54；34；10）波段合成的假彩色合成图；a2～j2为a1～j1中红框部分放大后得到的图像，包含建筑物与植被混合区域的感兴趣区，有丰富地物特征，用于对比不同方法的性能；a3～j3为残差图，用来展示算法结果与真实值之间的差异，颜色越深代表误差越小。GT.地面真值（ground truth），是真实的高分高光谱影像，用作所有方法的参考标准；下同
Figure 5. Fusion results on Washington DC Mall dataset
表 1 评价指标
Table 1. Evaluation indicators
指标名称 英文缩写 评测侧重 评测标准 均方根误差 RMSE 光谱+空间 越小越好 峰值信噪比 PSNR 光谱+空间 越大越好 相对全局自适应误差 ERGAS 光谱+空间 越小越好 光谱角相似度 SAM 光谱 越小越好 结构相似度 SSIM 空间 越接近1越好 
下载: 导出CSV
表 2 Washington DC Mall数据集的试验结果
Table 2. Experimental results on Washington DC Mall dataset
方法 RMSE PSNR ERGAS SAM SSIM 3D-CNN算法 2.4891 41.7845 0.4578 0.7589 0.9725 SSFCNN算法 9.4605 25.5572 1.8401 4.0395 0.9702 TF-Net算法 1.4460 42.5038 0.3429 0.6059 0.9765 SSR-Net算法 1.8652 39.6610 0.3349 0.5798 0.9740 MCT-Net算法 1.3961 46.1773 0.2515 0.4551 0.9854 DSPNet算法 0.7107 48.0420 0.1271 0.2569 0.9954 LGCT算法 1.3180 42.6773 0.2354 0.3782 0.9867 本研究方法 0.6213 49.2089 0.1131 0.2342 0.9984
下载: 导出CSV
表 3 Houston数据集的试验结果
Table 3. Experimental results on Houston dataset
方法 RMSE PSNR ERGAS SAM SSIM 3D-CNN算法 1.4258 43.3598 1.9687 4.4583 0.9894 SSFCNN算法 1.1875 44.9814 4.0530 4.4223 0.9829 TF-Net算法 1.4120 43.4773 1.9500 4.5806 0.9905 SSR-Net算法 1.0879 45.7422 1.7117 4.1521 0.9940 MCT-Net算法 0.9666 46.7697 1.5218 3.9865 0.9953 DSPNet算法 0.7754 48.6836 1.1525 3.5029 0.9971 LGCT算法 0.8194 48.2043 1.1966 3.5722 0.9970 本研究方法 0.7360 49.1366 1.1154 3.4844 0.9973
下载: 导出CSV
表 4 特征融合模块的消融试验
Table 4. Ablation experiments of feature fusion module
SSFTB MCB PSNR × √ 48.1299 √ × 48.2491 × × 47.8884 √ √ 49.1366 注：“×”表示去掉了该模块；“√”表示使用了该模块；SSFTB. 光谱−空间融合Transformer子模块；MCB. 多尺度卷积子模块；下同
下载: 导出CSV
表 5 特征增强模块的消融试验
Table 5. Ablation experiments of feature augmentation module
ATB DCB LACB PSNR √ × √ 48.7133 × × √ 48.7489 √ √ × 48.4699 √ √ √ 49.1366 注：ATB. 增强Transformer块；DCB. 深度卷积块；LACB. 局部特征增强子模块
下载: 导出CSV
-
[1] 李虹江, 于昕左, 马佳, 等. 多源遥感数据融合的高陡边坡危岩体信息提取[J]. 地质科技通报, 2025, 44(6): 306-316. doi: 10.19509/j.cnki.dzkq.tb20230695LI H J, YU X Z, MA J, et al. Information extraction of dangerous rock masses on high and steep slopes using multi-source remote sensing data fusion[J]. Bulletin of Geological Science and Technology, 2025, 44(6): 306-316. (in Chinese with English abstract) doi: 10.19509/j.cnki.dzkq.tb20230695 [2] 许伟, 杨德芳, 陈李昊, 等. 多源遥感影像融合处理提取格拉丹东雪山区域特征信息[J]. 地质科技通报, 2024, 43(2): 370-385. doi: 10.19509/j.cnki.dzkq.tb20220545XU W, YANG D F, CHEN L H, et al. Fusion processing of multisource remote sensing images for extracting characteristic information from the Geladandong Snow Mountain area[J]. Bulletin of Geological Science and Technology, 2024, 43(2): 370-385. (in Chinese with English abstract) doi: 10.19509/j.cnki.dzkq.tb20220545 [3] 尹政, 陈庆祥, 何剑波, 等. 基于多源融合建模和热−流耦合模拟的张掖盆地地热资源分布特征及成因机制[J]. 地质科技通报, 2025, 44(5): 3-12.YIN Z, CHEN Q X, HE J B, et al. Spatial characteristics and genetic mechanisms of geothermal resources in the Zhangye Basin based on multi-source fusion modeling and heat-flow coupling simulations[J]. Bulletin of Geological Science and Technology, 2025, 44(5): 3-12. (in Chinese with English abstract) [4] 李树涛, 李聪妤, 康旭东. 多源遥感图像融合发展现状与未来展望[J]. 遥感学报, 2021, 25(1): 148-166.LI S T, LI C Y, KANG X D. Development status and future prospects of multi-source remote sensing image fusion[J]. National Remote Sensing Bulletin, 2021, 25(1): 148-166. (in Chinese with English abstract) [5] VIVONE G. Multispectral and hyperspectral image fusion in remote sensing: A survey[J]. Information Fusion, 2023, 89: 405-417. doi: 10.1016/j.inffus.2022.08.032 [6] DIAN R W, LI S T, SUN B, et al. Recent advances and new guidelines on hyperspectral and multispectral image fusion[J]. Information Fusion, 2021, 69: 40-51. doi: 10.1016/j.inffus.2020.11.001 [7] DIAN R W, LI S T, FANG L Y, et al. Multispectral and hyperspectral image fusion with spatial-spectral sparse representation[J]. Information Fusion, 2019, 49: 262-270. doi: 10.1016/j.inffus.2018.11.012 [8] WU H J, WU S Q, ZHANG K F, et al. Unsupervised blind spectral-spatial cross-super-resolution network for HSI and MSI fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5511914. [9] YOKOYA N, YAIRI T, IWASAKI A. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2012, 50(2): 528-537. doi: 10.1109/TGRS.2011.2161320 [10] ZHANG Y F, DE BACKER S, SCHEUNDERS P. Noise-resistant wavelet-based Bayesian fusion of multispectral and hyperspectral images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2009, 47(11): 3834-3843. doi: 10.1109/TGRS.2009.2017737 [11] DIAN R W, LI S T, FANG L Y. Learning a low tensor-train rank representation for hyperspectral image super-resolution[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(9): 2672-2683. doi: 10.1109/TNNLS.2018.2885616 [12] REN C, HE X H, TENG Q Z, et al. Single image super-resolution using local geometric duality and non-local similarity[J]. IEEE Transactions on Image Processing, 2016, 25(5): 2168-2183. doi: 10.1109/TIP.2016.2542442 [13] YE F, WU Z B, XU Y, et al. Bayesian hyperspectral image super-resolution in the presence of spectral variability[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5545613. [14] WANG W Q, ZHANG J Q, LIU H, et al. Joint low-rank and sparse decomposition for infrared and visible image sequence fusion[J]. Infrared Physics & Technology, 2023, 133: 104828. doi: 10.1016/j.infrared.2023.104828 [15] MASI G, COZZOLINO D, VERDOLIVA L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7): 594. doi: 10.3390/rs8070594 [16] PALSSON F, SVEINSSON J R, ULFARSSON M O. Multispectral and hyperspectral image fusion using a 3-D-convolutional neural network[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(5): 639-643. doi: 10.1109/LGRS.2017.2668299 [17] ALAKWAA W, NASSEF M, BADR A. Lung cancer detection and classification with 3D convolutional neural network (3D-CNN)[J]. International Journal of Advanced Computer Science and Applications, 2017, 8(8). [18] LIU X Y, WANG Y H, LIU Q J. Remote sensing image fusion based on Two-stream fusion network[C]//MultiMedia Modeling. Cham: Springer, 2018: 428-439. [19] HAN X H, SHI B X, ZHENG Y Q. SSF-CNN: Spatial and spectral fusion with CNN for hyperspectral image super-resolution[C]//2018 25th IEEE International Conference on Image Processing (ICIP). Athens, Greece: IEEE, 2018: 2506-2510. [20] BANDARA W G C, PATEL V M. HyperTransformer: A textural and spectral feature fusion transformer for pansharpening[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA: IEEE, 2022: 1757-1767. [21] ZHANG X T, HUANG W, WANG Q, et al. SSR-NET: Spatial-spectral reconstruction network for hyperspectral and multispectral image fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7): 5953-5965. doi: 10.1109/TGRS.2020.3018732 [22] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30 (NeurIPS 2017). Long Beach, CA, USA: Curran Associates Inc. , 2017: 6000-6010. [23] LIU Z, LIN Y T, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada: IEEE, 2021: 9992-10002. [24] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C]//Proceedings of the 9th International Conference on Learning Representations (ICLR). Virtual Event: OpenReview, 2021. [25] WANG X H, WANG X Y, SONG R X, et al. MCT-Net: Multi-hierarchical cross transformer for hyperspectral and multispectral image fusion[J]. Knowledge-Based Systems, 2023, 264: 110362. doi: 10.1016/j.knosys.2023.110362 [26] SUN Y C, XU H, MA Y, et al. Dual spatial-spectral pyramid network with transformer for hyperspectral image fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5526016. [27] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017: 936-944. [28] HE W Q, FU X Y, LI N Y, et al. LGCT: Local-global collaborative transformer for fusion of hyperspectral and multispectral images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5537114. [29] SARA D, MANDAVA A K, KUMAR A, et al. Hyperspectral and multispectral image fusion techniques for high resolution applications: A review[J]. Earth Science Informatics, 2021, 14(4): 1685-1705. doi: 10.1007/s12145-021-00621-6 [30] YU H Y, LING Z X, ZHENG K, et al. Unsupervised hyperspectral and multispectral image fusion with deep spectral-spatial collaborative constraint[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5534114. doi: 10.1109/tgrs.2024.3472226 [31] LI J X, ZHENG K, YAO J, et al. Deep unsupervised blind hyperspectral and multispectral data fusion[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 6007305. [32] WOO S, PARK J, LEE J Y, et al. CBAM: Convolutional block attention module[C]//Computer Vision-ECCV 2018. Munich, Germany: Springer, 2018: 3-19. [33] ZHANG Y H, YAN J C. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting[C]//International Conference on Learning Representations (ICLR 2023). Kigali, Rwanda: OpenReview, 2023. [34] AUDEBERT N, LE SAUX B, LEFEVRE S. Deep learning for classification of hyperspectral data: A comparative review[J]. IEEE Geoscience and Remote Sensing Magazine, 2019, 7(2): 159-173. doi: 10.1109/MGRS.2019.2912563 [35] DADRASS JAVAN F, SAMADZADEGAN F, MEHRAVAR S, et al. A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171: 101-117. doi: 10.1016/j.isprsjprs.2020.11.001 [36] YOKOYA N, GROHNFELDT C, CHANUSSOT J. Hyperspectral and multispectral data fusion: A comparative review of the recent literature[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(2): 29-56. doi: 10.1109/MGRS.2016.2637824 [37] HUYNH-THU Q, GHANBARI M. Scope of validity of PSNR in image/video quality assessment[J]. Electronics Letters, 2008, 44(13): 800-801. doi: 10.1049/el:20080522 [38] WALD L. Quality of high resolution synthesised images: Is there a simple criterion[C]//Proceedings of the Third International Conference Fusion of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images. Nice, France: SEE/URISCA, 2000: 99-103. [39] CHANG Y, YAN L X, ZHONG S. Hyper-Laplacian regularized unidirectional low-rank tensor recovery for multispectral image denoising[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE Computer Society, 2017: 5901-5909. [40] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. doi: 10.1109/TIP.2003.819861 -
下载:
点击查看大图
计量
- 文章访问数: 356
- PDF下载量: 50
- 被引次数: 0
