Deep Learning-Based Image Fusion: A Systematic Literature Review on Trends, Datasets, Evaluation Methods, and Research Challenges

Raza Haan Fiddo Aryasturangga; Pangki Deswa Rayhandi; Kholis

doi:10.59934/jaiea.v5i3.2506

Authors

Raza Haan Fiddo Aryasturangga Universitas Majalengka
Pangki Deswa Rayhandi Universitas Majalengka
Kholis Universitas Majalengka

DOI:

https://doi.org/10.59934/jaiea.v5i3.2506

Keywords:

Convolutional Neural Network; Deep Learning; Digital Image Processing; Image Fusion; Systematic Literature Review

Abstract

Advances in digital image processing technology have increased the need for image fusion techniques to produce more accurate and high-quality visual information. This study aims to analyze methodological developments, research trends, datasets, evaluation methods, and challenges in image fusion research through a Systematic Literature Review (SLR) using the PRISMA framework. The literature selection process resulted in 335 papers being included for analysis. The findings indicate that deep learning-based methods, particularly Convolutional Neural Networks (CNNs), have become the dominant approach in modern image fusion research. Furthermore, recent studies have increasingly explored Generative Adversarial Networks (GANs), Transformers, and hybrid methods to improve fusion performance. The most significant challenges identified include the need for large-scale datasets, high computational complexity, the lack of standardized evaluation frameworks, and limitations in model generalization. These findings provide a comprehensive overview of current developments and highlight future research opportunities in deep learning-based image fusion.

Downloads

Download data is not yet available.

References

C. Pohl and J. L. Van Genderen, “Review article Multisensor image fusion in remote sensing: Concepts, methods and applications,” 1998. doi: 10.1080/014311698215748.

Z. Zhang and R. S. Blum, “A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application,” Proceedings of the IEEE, vol. 87, pp. 1315–1326, 1999, doi: 10.1109/5.775414.

H. Li and X. J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, pp. 2614–2623, May 2019, doi: 10.1109/TIP.2018.2887342.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, pp. 600–612, Apr. 2004, doi: 10.1109/TIP.2003.819861.

J. Ma, Y. Ma, and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, Jan. 2019, doi: 10.1016/j.inffus.2018.02.004.

Y. Liu, X. Chen, J. Cheng, H. Peng, and Z. Wang, “Infrared and visible image fusion with convolutional neural networks,” vol. 16, no. 3, 2018.

B. C. S. Kitchenham, “Guidelines for Performing Systematic Literature Reviews in Software Engineering,” 2007.

C. C. Szeto and E. Hung, “Mining outliers with faster cutoff update and space utilization,” Pattern Recognit. Lett., vol. 31, pp. 1292–1301, Aug. 2010, doi: 10.1016/j.patrec.2010.04.002.

L. Yang, S. Ding, F. Zhou, X. Yang, and Y. Xiao, “Robust EEG feature learning model based on an adaptive weight and pairwise-fused LASSO,” Biomed. Signal Process. Control, vol. 68, Jul. 2021, doi: 10.1016/j.bspc.2021.102728.

H. Wang, H. Jiang, and K. Gong, “ML-Based Forward Symbol Timing Offset Estimation for Burst Signals,” IEEE Access, vol. 8, pp. 200875–200882, 2020, doi: 10.1109/ACCESS.2020.3035733.

C. Boutsidis and E. Gallopoulos, “SVD based initialization: A head start for nonnegative matrix factorization,” Pattern Recognit., vol. 41, pp. 1350–1362, Apr. 2008, doi: 10.1016/j.patcog.2007.09.010.

Y. Liu, X. Chen, H. Peng, and Z. Wang, “Multi-focus image fusion with a deep convolutional neural network,” Information Fusion, vol. 36, pp. 191–207, Jul. 2017, doi: 10.1016/j.inffus.2016.12.001.

H. M. Jawad, R. Nordin, S. K. Gharghan, A. M. Jawad, M. Ismail, and M. J. Abu-Alshaeer, “Power reduction with sleep/wake on redundant data (SWORD) in a wireless sensor network for energy-efficient precision agriculture,” Sensors (Switzerland), vol. 18, Oct. 2018, doi: 10.3390/s18103450.

Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao, and L. Zhang, “IFCNN: A general image fusion framework based on convolutional neural network,” Information Fusion, vol. 54, pp. 99–118, Feb. 2020, doi: 10.1016/j.inffus.2019.07.011.

Z. Zhou, S. Li, and B. Wang, “Multi-scale weighted gradient-based fusion for multi-focus images,” Information Fusion, vol. 20, pp. 60–72, 2014, doi: 10.1016/j.inffus.2013.11.005.

L. Huang, X. Liu, Y. Liu, B. Lang, and D. Tao, “Centered Weight Normalization in Accelerating Training of Deep Neural Networks,” in Proceedings of the IEEE International Conference on Computer Vision, Institute of Electrical and Electronics Engineers Inc., Dec. 2017, pp. 2822–2830. doi: 10.1109/ICCV.2017.305.

Q. Wang, S. Li, H. Qin, and A. Hao, “Robust multi-modal medical image fusion via anisotropic heat diffusion guided low-rank structural analysis,” Information Fusion, vol. 26, pp. 103–121, May 2015, doi: 10.1016/j.inffus.2015.01.001.

H. Zhang, H. Xu, X. Tian, J. Jiang, and J. Ma, “Image fusion meets deep learning: A survey and perspective,” Dec. 2021, Elsevier B.V. doi: 10.1016/j.inffus.2021.06.008.

T. H. N. Le, Y. Zheng, C. Zhu, K. Luu, and M. Savvides, “Multiple Scale Faster-RCNN Approach to Driver’s Cell-Phone Usage and Hands on Steering Wheel Detection,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, IEEE Computer Society, Dec. 2016, pp. 46–53. doi: 10.1109/CVPRW.2016.13.

D. Fourure, R. Emonet, E. Fromont, D. Muselet, A. Tremeau, and C. Wolf, “Residual conv-deconv grid network for semantic segmentation,” in British Machine Vision Conference 2017, BMVC 2017, BMVA Press, 2017. doi: 10.5244/c.31.181.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, Dec. 2016, pp. 770–778. doi: 10.1109/CVPR.2016.90.

S. Li, X. Kang, L. Fang, J. Hu, and H. Yin, “Pixel-level image fusion: A survey of the state of the art,” Information Fusion, vol. 33, pp. 100–112, Jan. 2017, doi: 10.1016/j.inffus.2016.05.004.

H. Xu, J. Ma, Z. Le, J. Jiang, and X. Guo, “FusionDN: A unified densely connected network for image fusion,” in AAAI 2020 - 34th AAAI Conference on Artificial Intelligence, AAAI press, 2020, pp. 12484–12491. doi: 10.1609/aaai.v34i07.6936.

P. J. Burt and E. H. Adelson, “The Laplacian Pyramid as a Compact Image Code,” IEEE Transactions on Communications, vol. 31, pp. 532–540, 1983, doi: 10.1109/TCOM.1983.1095851.

S. Li, X. Kang, and J. Hu, “Image fusion with guided filtering,” IEEE Transactions on Image Processing, vol. 22, pp. 2864–2875, 2013, doi: 10.1109/TIP.2013.2244222.

H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, “U2Fusion: A Unified Unsupervised Image Fusion Network,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, pp. 502–518, Jan. 2022, doi: 10.1109/TPAMI.2020.3012548.

Y. Liu, X. Chen, J. Cheng, H. Peng, and Z. Wang, “Infrared and visible image fusion with convolutional neural networks,” vol. 16, no. 3, 2018.

C. Pohl and J. L. Van Genderen, “Review article Multisensor image fusion in remote sensing: Concepts, methods and applications,” 1998. doi: 10.1080/014311698215748.

H. Li and X. J. Wu, “DenseFuse: A fusion approach to infrared and visible images,” IEEE Transactions on Image Processing, vol. 28, pp. 2614–2623, May 2019, doi: 10.1109/TIP.2018.2887342.

N. R. Haddaway, M. J. Page, C. C. Pritchard, and L. A. McGuinness, “PRISMA2020: An R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis,” Campbell Systematic Reviews, vol. 18, Jun. 2022, doi: 10.1002/cl2.1230.

H. Snyder, “Literature review as a research methodology: An overview and guidelines,” J. Bus. Res., vol. 104, pp. 333–339, Nov. 2019, doi: 10.1016/j.jbusres.2019.07.039.

D. Tranfield, D. Denyer, and P. Smart, “Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review,” 2003. doi: 10.1111/1467-8551.00375.

B. C. S. Kitchenham, “Guidelines for Performing Systematic Literature Reviews in Software Engineering,” 2007.

M. Nejati, S. Samavi, and S. Shirani, “Multi-focus image fusion using dictionary-based sparse representation,” Information Fusion, vol. 25, pp. 72–84, 2015, doi: 10.1016/j.inffus.2014.10.004.

Z. Zhang and R. S. Blum, “A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application,” Proceedings of the IEEE, vol. 87, pp. 1315–1326, 1999, doi: 10.1109/5.775414.

J. Ma, Y. Ma, and C. Li, “Infrared and visible image fusion methods and applications: A survey,” Information Fusion, vol. 45, pp. 153–178, Jan. 2019, doi: 10.1016/j.inffus.2018.02.004.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, pp. 600–612, Apr. 2004, doi: 10.1109/TIP.2003.819861.