Please wait a minute...
Computational Visual Media  2019, Vol. 05 Issue (04): 391-401    doi: 10.1007/s41095-019-0158-8
Research Article     
Adaptive deep residual network for single image super-resolution
Shuai Liu1, Ruipeng Gang1, Chenghua Li2,(✉), Ruixia Song1
1North China University of Technology, Beijing, 100043, China. E-mail: S. Liu, 18601200232@163.com; R. Gang, gang_rp909@sina.com; R. Song, songrx@ncut.edu.cn.;
2Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China.
Download: PDF (17705 KB)      HTML  
Export: BibTeX | EndNote (RIS)      

Abstract  

In recent years, deep learning has achieved great success in the field of image processing. In the single image super-resolution (SISR) task, the con-volutional neural network (CNN) extracts the features of the image through deeper layers, and has achieved impressive results. In this paper, we propose a singleimage super-resolution model based on Adaptive Deep Residual named as ADR-SR, which uses the Input Output Same Size (IOSS) structure, and releases the dependence of upsampling layers compared with the existing SR methods. Specifically, the key element of our model is the Adaptive Residual Block (ARB), which replaces the commonly used constant factor with an adaptive residual factor. The experiments prove the effectiveness of our ADR-SR model, which can not only reconstruct images with better visual effects, but also get better objective performances.



Key wordssingle image super-resolution (SISR)      adaptive deep residual network      deep learning     
Received: 15 December 2019      Published: 13 March 2020
Corresponding Authors: Chenghua Li   
About author:

* Shuai Liu and Ruipeng Gang contributed equally to this work.

Cite this article:

Shuai Liu, Ruipeng Gang, Chenghua Li, Ruixia Song. Adaptive deep residual network for single image super-resolution. Computational Visual Media, 2019, 05(04): 391-401.

URL:

http://cvm.tsinghuajournals.com/10.1007/s41095-019-0158-8     OR     http://cvm.tsinghuajournals.com/Y2019/V05/I04/391

Fig. 1:  Comparison of (a) EDSR-baseline structure with our (b) ADR-SR structure. Note that our ADR-SR does not have any upsampling layers and uses Adaptive Residual Block (ARB). The position of the global residuals is modified, and the depth and width of the network are also modified.
Fig. 2:  Comparison of input and output between the first type and the second type of network. (a) The first type of network, which has different input and output sizes, reconstructs 4 output pixels from 1 input pixel. (b) The second type of network, which has the same input and output size, reconstructs 1 output pixel from 1 input pixel.
Fig. 3:  Effects of adding SE module in different positions on the model.
Fig. 4:  Comparison of different residual block structures. (a) Original residual block, which is proposed in ResNet. (b) SRResNet residual block, which removes the last activation function from the original residual block. (c) EDSR residual block, which removes the BatchNormalization from the SRResNet residual block and adds a fixed factor of 0.1. (d) Ours Adaptive Residual Block (ARB), which replaces the fixed factor with the SE modules to increase adaptability.
Fig. 5:  Effects of increasing depth and increasing width on the model.
ScaleMethodSet5Set14B100U100DIV2K Val
PSNRSSIMPSNRSSIMPSNRSSIMPSNRSSIMPSNRSSIM
×2LapSRN34.800.93629.600.86925.920.79026.830.86833.340.925
VDSR34.960.93630.530.88029.580.87028.530.89333.450.925
DRRN35.010.93630.610.88229.900.87728.880.89833.620.927
CARN35.620.94230.150.88125.880.79327.770.88634.500.936
MCAN35.800.94330.260.88225.850.79228.030.88834.580.937
EDSR baseline35.780.94330.190.88125.890.79327.820.88734.620.937
ADR-SR35.820.94331.410.89330.120.88030.040.91434.640.937
×4LapSRN29.390.84925.300.69924.830.67722.220.68828.170.799
VDSR28.900.83225.860.70725.860.70723.230.70327.920.784
SRResNet29.760.85525.360.70424.830.67922.210.69628.100.806
SRDenseNet29.030.83825.700.71325.680.69523.540.72527.960.790
DRRN29.210.83826.040.71325.660.68323.540.71728.070.789
CARN30.230.86425.660.71224.980.68422.720.71128.960.818
MCAN30.420.86525.730.71424.940.68522.900.71829.030.820
EDSR baseline30.180.86325.650.71224.960.68422.690.71128.960.818
ADR-SR30.210.86326.710.74126.080.70624.480.76028.940.817
Table 1: Quantitative comparison with the state-of-the-art methods based on ×2, ×4 SR with bicubic degradation model
Fig. 6:  Comparison of experimental effects of Urban100 dataset.
Fig. 7:  Comparison of experimental effects of DIV2K validation dataset.
[1]   Su, B.; Jin, W. POCS-MPMAP based super-resolution image restoration. Acta Photonica Sinica Vol. 32, No. 4, 502-504, 2003.
[2]   Choi, J.-S.; Kim, M. Single image super-resolution using global regression based on multiple local linear mappings. IEEE Transactions on Image Processing Vol. 26, No. 3, 1300-1314, 2017.
[3]   Yang, J. C.; Wang, Z. W.; Lin, Z.; Cohen, S.; Huang, T. Coupled dictionary training for image super-resolution. IEEE Transactions on Image Processing Vol. 21, No. 8, 3467-3478, 2012.
[4]   Yang, J. C.; Wright, J.; Huang, T. S.; Ma, Y. Image super-resolution via sparse representation. IEEE Transactions on Image Processing Vol. 19, No. 11, 2861-2873, 2010.
[5]   Zhang, F. L.; Wang, J.; Shechtman, E.; Zhou, Z. Y.; Shi, J. X.; Hu, S. M. PlenoPatch: Patch-based plenoptic image manipulation. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 5, 1561-1573, 2017.
[6]   Barnes, C.; Zhang, F.-L. A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media Vol. 3, No. 1, 3-20, 2017.
[7]   Liu, H.; Fu, Z. L.; Han, J. G.; Shao, L.; Hou, S. D.; Chu, Y. Z. Single image super-resolution using multi-scale deep encoder-decoder with phase congruency edge map guidance. Information Sciences Vol. 473, 44-58,2019.
[8]   Krizhevsky, A.; Sutskever, I.; Hinton, G. E. ImageNet classification with deep convolutional neural networks. In: Proceedings of the Advances in Neural Information Processing Systems 25, 1097-1105, 2012.
[9]   Shelhamer, E.; Long, J.; Darrell, T. Fully con-volutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 39, No. 4, 640-651, 2017.
[10]   Girshick, R.; Donahue, J.; Darrell, T.; Malik, J.Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 580-587, 2014.
[11]   Lim, B.; Son, S.; Kim, H.; Nah, S.; Lee, K. M.Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 136-144, 2017.
[12]   Ledig, C.; Theis, L.; Huszár, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Tota, J.; Wang, Z.; Shi, W.Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4681-4690, 2017.
[13]   Lai, W.-S.; Huang, J.-B.; Ahuja, N.; Yang, M.-H.Deep Laplacian Pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 624-632, 2017.
[14]   Dong, C.; Loy, C. C.; He, K. M.; Tang, X. O. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 38, No. 2, 295-307, 2016.
[15]   Tai, Y.; Yang, J.; Liu, X. M.Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3147-3155, 2017.
[16]   Kim, J.; Lee, J. K.; Lee, K. M.Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1646-1654, 2016.
[17]   He, K. M.; Zhang, X. Y.; Ren, S. Q.; Sun, J.Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision, 1026-1034, 2015.
[18]   Szegedy, C.; Liu, W.; Jia, Y. Q.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A.Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1-9, 2015.
[19]   Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; Polosukhin, I.Attention is all you need. In: Proceedings of the Advances in Neural Information Processing Systems 30, 5998-6008, 2017.
[20]   Bell, S.; Zitnick, C. L.; Bala, K.; Girshick, R.Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2874-2883, 2016.
[21]   Hu, J.; Shen, L.; Sun, G.Squeeze-and-excitation net-works. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7132-7141, 2018.
[22]   Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S. A.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; Fei-Fei, L. ImageNet large scale visual recognition challenge. International Journal of Computer Vision Vol. 115, No. 3, 211-252, 2015.
[23]   Agustsson, E.; Timofte, R.NTIRE 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 126-135, 2017.
[24]   Bevilacqua, M.; Roumy, A.; Guillemot, C.; Alberi Morel, M.-L.Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the 23rd British Machine Vision Conference, 135.1-135.10, 2012.
[25]   Zeyde, R.; Elad, M.; Protter, M.On single image scale-up using sparse-representations. In: Curves and Surfaces 2010. Lecture Notes in Computer Science, Vol. 6920. Boissonnat, J. D. et al. Eds. Springer Berlin Heidelberg, 711-730, 2012.
[26]   Martin, D.; Fowlkes, C.; Tal, D.; Malik, J.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the 8th IEEE International Conference on Computer Vision, Vol. 2, 416-423, 2001.
[27]   Huang, J.-B.; Singh, A.; Ahuja, N.Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5197-5206, 2015.
[28]   Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[29]   Tong, T.; Li, G.; Liu, X. J.; Gao, Q. Q.Image super-resolution using dense skip connections. In: Proceedings of the IEEE International Conference on Computer Vision, 4799-4807, 2017.
[30]   Ahn, N.; Kang, B.; Sohn, K.-A.Fast, accurate, and lightweight super-resolution with cascading residual network. In: Computer Vision - ECCV 2018. Lecture Notes in Computer Science, Vol. 11214. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 256-272, 2018.
[31]   Ma, H.; Chu, X.; Zhang, B.; Wan, S. A matrix-in-matrix neural network for image super resolution. arXiv preprint arXiv:1903.07949, 2019.
[1] Yoshikatsu Nakajima,Hideo Saito. Robust camera pose estimation by viewpoint classification using deep learning[J]. Computational Visual Media, 2017, 3(2): 189-198.