View source for Multi-focus image fusion

{{Short description|Combining multiple images to create one}}
'''Multi-focus image fusion''' is a [[image compression|multiple image compression]] technique using input images with different [[Depth of focus|focus depths]] to make one output image that preserves all information.

== Overview ==
In recent years, [[image fusion]] has been used in many applications such as remote sensing, [[surveillance]], medical diagnosis, and photography applications.
Two major applications of image fusion in photography are fusion of multi-focus images and [[Multi-Exposure|multi-exposure]] images.<ref name=":0" /><ref name=":4">{{Cite journal|last1=Li|first1=Shutao|last2=Kang|first2=Xudong|last3=Fang|first3=Leyuan|last4=Hu|first4=Jianwen|last5=Yin|first5=Haitao|date=2017-01-01|title=Pixel-level image fusion: A survey of the state of the art|journal=Information Fusion|volume=33|pages=100–112|doi=10.1016/j.inffus.2016.05.004|s2cid=9263669 |issn=1566-2535}}</ref> 

The main idea of image fusion is gathering important and the essential information from the input images into one single image which ideally has all of the information of the input images.<ref name=":0">{{Cite journal|last1=Amin-Naji|first1=Mostafa|last2=Aghagolzadeh|first2=Ali|last3=Ezoji|first3=Mehdi|date=2019|title=Ensemble of CNN for multi-focus image fusion|journal=Information Fusion|volume=51|pages=201–214|doi=10.1016/j.inffus.2019.02.003|s2cid=150059597 |issn=1566-2535}}</ref><ref name=":1">{{Cite journal|last1=Amin-Naji|first1=Mostafa|last2=Aghagolzadeh|first2=Ali|last3=Ezoji|first3=Mehdi|date=2019|title=CNNs hard voting for multi-focus image fusion|journal=Journal of Ambient Intelligence and Humanized Computing|volume=11|issue=4|pages=1749–1769|doi=10.1007/s12652-019-01199-0|s2cid=86563059 |issn=1868-5145}}</ref><ref name=":5">{{Cite journal|last1=Liu|first1=Yu|last2=Chen|first2=Xun|last3=Peng|first3=Hu|last4=Wang|first4=Zengfu|date=2017-07-01|title=Multi-focus image fusion with a deep convolutional neural network|journal=Information Fusion|volume=36|pages=191–207|doi=10.1016/j.inffus.2016.12.001|s2cid=11925688 |issn=1566-2535}}</ref><ref name=":2">{{Cite journal|last1=Amin-Naji|first1=Mostafa|last2=Aghagolzadeh|first2=Ali|date=2018|title=Multi-Focus Image Fusion in DCT Domain using Variance and Energy of Laplacian and Correlation Coefficient for Visual Sensor Networks|journal=Journal of AI and Data Mining|volume=6|issue=2|pages=233–250|doi=10.22044/jadm.2017.5169.1624|issn=2322-5211}}</ref> The research history of image fusion spans over 30 years and many scientific papers.<ref name=":4" /><ref name=":6">{{Cite journal|last1=Liu|first1=Yu|last2=Chen|first2=Xun|last3=Wang|first3=Zengfu|last4=Wang|first4=Z. Jane|author4-link=Zhen Jane Wang|last5=Ward|first5=Rabab K.|last6=Wang|first6=Xuesong|date=2018-07-01|title=Deep learning for pixel-level image fusion: Recent advances and future prospects|journal=Information Fusion|volume=42|pages=158–173|doi=10.1016/j.inffus.2017.10.007|s2cid=46849537 |issn=1566-2535}}</ref> Image fusion generally has two aspects: image fusion methods and objective evaluation metrics.<ref name=":6" /> 
[[File:Sample of Multi-Focus image fusion.png|thumb|475x475px|'''A Sample of Multi-Focus Image Fusion''']]
In [[Visual sensor network|visual sensor networks (VSN)]], sensors are cameras which record images and video sequences. In many applications of VSN, a camera can't give a perfect [[illustration]] including all details of the scene. This is because of the limited depth of focus of the [[optical lens]] of cameras. Therefore, just the object located in the [[focal length]] of camera is focused and clear, and other parts of the image are blurred. 

VSN captures images with different depths of focus using several cameras. Due to the large amount of data generated by cameras compared to other sensors such as pressure and temperature sensors and some limitations of [[Bandwidth (signal processing)|bandwidth]], energy consumption and processing time, it is essential to process the local input images to decrease the amount of transmitted data.<ref name=":4" />

Much research on multi-focus image fusion has been done in recent years and can be classified into two categories: transform and spatial domains. Commonly used transforms for image fusion are [[Discrete cosine transform]] (DCT) and Multi-Scale Transform (MST).<ref name=":4" /> <ref name=":7">{{Cite journal|last1=Haghighat|first1=Mohammad Bagher Akbari|last2=Aghagolzadeh|first2=Ali|last3=Seyedarabi|first3=Hadi|date=2011-09-01|title=Multi-focus image fusion for visual sensor networks in DCT domain|journal=Computers & Electrical Engineering|series=Special Issue on Image Processing|volume=37|issue=5|pages=789–797|doi=10.1016/j.compeleceng.2011.04.016|s2cid=38131177 |issn=0045-7906}}</ref> Recently, [[Deep learning]] (DL) has been thriving in several image processing and [[computer vision]] applications.<ref name=":0" /><ref name=":1" /><ref name=":3">{{Cite book|last1=Amin-Naji|first1=Mostafa|last2=Aghagolzadeh|first2=Ali|last3=Ezoji|first3=Mehdi|title=2018 9th International Symposium on Telecommunications (IST) |chapter=Fully Convolutional Networks for Multi-Focus Image Fusion |date=2018|pages=553–558|doi=10.1109/ISTEL.2018.8660989|isbn=978-1-5386-8274-6|s2cid=71150698 }}</ref>

== Multi-Focus image fusion in the spatial domain ==
Huang and Jing have reviewed and applied several focus measurements in the spatial domain for the multi-focus image fusion process,  suitable for real-time applications. They mentioned some focus measurements including [[variance]], energy of [[image gradient]] (EOG), Tenenbaum's algorithm (Tenengrad), energy of [[Laplace operator|Laplacian]] (EOL), sum-modified-[[Laplacian]] (SML), and [[spatial frequency]] (SF). Their experiments showed that EOL gave better results than other methods like variance and spatial frequency.<ref>{{Cite journal|last1=Huang|first1=Wei|last2=Jing|first2=Zhongliang|date=2007-03-01|title=Evaluation of focus measures in multi-focus image fusion|journal=Pattern Recognition Letters|volume=28|issue=4|pages=493–500|doi=10.1016/j.patrec.2006.09.005|bibcode=2007PaReL..28..493H |issn=0167-8655}}</ref><ref name=":2" />

== Multi-Focus image fusion in multi-scale transform and DCT domain ==
Image fusion based on the multi-scale transform is the most commonly used and promising technique. Laplacian [[pyramid (image processing)|pyramid]] transform, gradient pyramid-based transform, morphological pyramid transform and the premier ones, discrete [[wavelet transform]], [[shift-invariant system|shift-invariant]] wavelet transform (SIDWT), and [[discrete cosine transform|discrete cosine]] [[harmonic wavelet transform]] (DCHWT) are some examples of image fusion methods based on multi-scale transform.<ref name=":4" /><ref name=":2" /> <ref name=":7" /> These methods are complex and have some limitations e.g. processing time and energy consumption. For example, multi-focus image fusion methods based on DWT require a lot of [[convolution]] operations, so they take more time and energy to process. Therefore, most methods in multi-scale transform are not suitable for real-time applications.<ref name=":7" /><ref name=":2" /> Moreover, these methods are not very successful along edges, due to the [[wavelet]] transform process missing the edges of the image. They create ringing [[Artifact (error)|artefacts]] in the output image and reduce its quality. 

Due to the aforementioned problems in the multi-scale transform methods, researchers are interested in multi-focus image fusion in the DCT domain. DCT-based methods are more efficient in terms of transmission and archiving images coded in [[Joint Photographic Experts Group]] (JPEG) standard to the upper node in the VSN agent. A JPEG system consists of a pair of an [[Encoder (digital)|encoder]] and a decoder. In the encoder, images are divided into non-overlapping 8×8 blocks, and the DCT [[Coefficient|coefficients]] are calculated for each. Since the quantization of DCT coefficients is a [[Lossy compression|lossy]] process, many of the small-valued DCT coefficients are quantized to zero, which corresponds to high frequencies. DCT-based image fusion algorithms work better when the multi-focus image fusion methods are applied in the compressed domain.<ref name=":7" /><ref name=":2" /> 

In addition, in the spatial-based methods, the input images must be decoded and then transferred to the spatial domain. After implementation of the image fusion operations, the output fused images must again be encoded. DCT domain-based methods do not require complex and time-consuming consecutive [[Decoding methods|decoding]] and encoding operations. Therefore, the image fusion methods based on DCT domain operate with much less energy and processing time.<ref name=":7" /><ref name=":2" /> Recently, a lot of research has been carried out in the DCT domain. DCT+Variance, DCT+Corr_Eng, DCT+EOL, and DCT+VOL are some prominent examples of DCT based methods.<ref name=":2" /><ref name=":7" />

== Multi-Focus image fusion using Deep Learning ==
Nowadays, the deep learning is utilized in image fusion applications such as multi-focus image fusion. Liu et al. were the first researchers that used CNN for multi-focus image fusion. They used the Siamese architecture for comparing the focused and unfocused patches.<ref name=":5" /> C. Du et al. submitted MSCNN method that obtains the initial segmented decision map with [[image segmentation]] between the focused and unfocused patches through the multi-scale convolution [[neural network]].<ref>{{Cite journal|last1=Du|first1=C.|last2=Gao|first2=S.|date=2017|title=Image Segmentation-Based Multi-Focus Image Fusion Through Multi-Scale Convolutional Neural Network|journal=IEEE Access|volume=5|pages=15750–15761|doi=10.1109/ACCESS.2017.2735019|s2cid=9466474 |doi-access=free|bibcode=2017IEEEA...515750D }}</ref> H. Tang et al. introduced the [[pixel]]-wise convolution neural network (p-CNN) for classification of the focused and unfocused patches.<ref>{{Cite journal|last1=Tang|first1=Han|last2=Xiao|first2=Bin|last3=Li|first3=Weisheng|last4=Wang|first4=Guoyin|date=2018-04-01|title=Pixel convolutional neural network for multi-focus image fusion|journal=Information Sciences|volume=433-434|pages=125–141|doi=10.1016/j.ins.2017.12.043|issn=0020-0255}}</ref> 

All of these CNN based multi-focus image fusion methods have enhanced the decision map. Nevertheless, their initial segmented decision maps have a lot of weakness and errors. Therefore, satisfaction of their final fusion decision map depends to use vast post-processing algorithms such as Consistency Verification (CV), [[Mathematical morphology|morphological]] operations, watershed, guiding filters, and small region removal on the initial segmented decision map. Along with the CNN based multi-focus image fusion methods, fully [[Convolutional neural network|convolutional]] network (FCN) is also utilized in multi-focus image fusion.<ref name=":3" /><ref>{{Cite journal|last1=Guo|first1=Xiaopeng|last2=Nie|first2=Rencan|last3=Cao|first3=Jinde|last4=Zhou|first4=Dongming|last5=Qian|first5=Wenhua|date=2018-06-12|title=Fully Convolutional Network-Based Multifocus Image Fusion|journal=Neural Computation|volume=30|issue=7|pages=1775–1800|doi=10.1162/neco_a_01098|pmid=29894654|s2cid=48358558 |issn=0899-7667}}</ref> 

=== ECNN: Ensemble of CNN for Multi-Focus Image Fusion<ref name=":0" /> ===
[[File:The schematic diagram of generating three datasets according to the proposed patch feeding.png|thumb|479x479px|The schematic diagram of generating three datasets according to the proposed patch feeding which is used in the training procedure of ECNN <ref name=":0" />]]
The [[Convolutional neural network|Convolutional Neural Networks]] (CNNs) based multi-focus image fusion methods have recently attracted enormous attention. They greatly enhanced the constructed decision map compared with the previous state of the art methods that have been done in the spatial and transform domains. Nevertheless, these methods have not reached to the satisfactory initial decision map, and they need to undergo vast post-processing [[Algorithm|algorithms]] to achieve a satisfactory decision map. 

In the method of ECNN, a novel CNNs based method with the help of the [[ensemble learning]] is proposed. It is very reasonable to use various models and datasets rather than just one. The ensemble learning based methods intend to pursue increasing diversity among the models and datasets in order to decrease the problem of the overfitting on the training [[Data set|dataset]]. 

It is obvious that the results of an ensemble of CNNs are better than just one single CNNs. Also, the proposed method introduces a new simple type of multi-focus images dataset. It simply changes the arranging of the patches of the multi-focus datasets, which is very useful for obtaining the better accuracy. With this new type arrangement of datasets, the three different datasets including the original and the [[Gradient]] in directions of vertical and horizontal patches are generated from the [[COCO (dataset)|COCO]] dataset. Therefore, the proposed method introduces a new network that three CNNs models which have been trained on three different created datasets to construct the initial segmented decision map. These ideas greatly improve the initial segmented decision map of the proposed method which is similar, or even better than, the other final decision map of CNNs based methods obtained after applying many post-processing algorithms. Many real multi-focus test images are used in our experiments, and the results are compared with quantitative and qualitative criteria. The obtained experimental results indicate that the proposed CNNs based network is more accurate and have the better decision map without post-processing algorithms than the other existing state of the art multi-focus fusion methods which used many post-processing algorithms.
[[File:ECNN flowhart of fusion of two images.png|thumb|610x610px|The flowchart of the proposed method of ECNN for getting the initial segmented decision map of multi-focus image fusion <ref name=":0" />|alt=]]
This method introduces a new network for achieving the cleaner initial segmented decision map compared with the others. The pro- posed method introduces a new architecture which uses an ensemble of three CNNs trained on three different datasets. Also, the proposed method prepares a new simple type of multi- focus image datasets for achieving the better fusion performance than the other popular multi-focus image datasets. 

This idea is very helpful to achieve the better initial segmented decision map, which is the same or even better than the others initial segmented decision map by using vast post-processing algorithms.  

[[File:ECNN Network.png|thumb|589x589px|The [[schematic]] of the proposed ECNN architecture with all details of models of CNNs <ref name=":0" />|alt=|center]]

== External links ==
The source code of ECNN http://amin-naji.com/publications/ and https://github.com/mostafaaminnaji/ECNN

== References ==
<references />

[[Category:Image processing]]
[[Category:Image compression]]