Simultaneous Smoothing and Sharpening of Color Images

New model that enable the smoothing of the noise at the same time that enlarge the details.

1. Simultaneous sharpening and smoothing of color images C. Pérez-Benito, S. Morillas, C. Jordán, J.A. Conejero Mathematical Modelling in Engineering & Human Behaviour Universitat Politècnica de València. July 2017

2. Introduction Colour image smoothing and sharpening are two important pre-processing techniques within the Computer Vision field. Smoothing consist on the removal of possible image perturbations resulted from the image acquisition. On the other hand, sharpening is in charge of the improvement of the image visual appearance and the enhancement the details and borders of the image. There exists a lot of smoothing and sharpening methods that are able to improve visual quality of images. However the same does not happen with the simultaneous approaches due to the opposite nature of these two operations. W e present a new model based on graph theory that allows us to improve the details of the image at the same time that the noise is removed.

3. Contents Sharpening vs smoothing Graph based model for color image processing Application of the model to simultaneous sharpening and smoothing Conclusion

4. Contents Sharpening vs smoothing Graph based model for color image processing Application of the model to simultaneous sharpening and smoothing Conclusions

5. Colour image smoothing is the set of pre-processing techniques intended for jk removing possible image perturbations without losing image information. White Gaussian noise is the most common factor which can significantly affect visual quality of images: each pixel of the image will be changed from its original value by some small amount that follows a Gaussian distribution. Colour image sharpening is the set of techniques whose purpose is the improvement of the image visual appearance and highlight or recover certain details of the image for conducting an suitable analysis by a human or a machine. Sharpening vs smoothing

6. Typical spatial filters for colour image smoothing are based on the convolution of the image with different kernels , depending on the intended result . These kernels could be of any size n x n, but usually 3 x 3 According to previous works, using n > 3 results in higher noise smoothing capability but much more blurred images, along with the large increase in computational cost that make increasing the window not a good choice in general. Sharpening vs smoothing

7. Sharpening vs smoothing Typical spatial filters for colour image smoothing are based on the convolution of the image with different kernels , depending on the intended result . These kernels could be of any size n x n, but usually 3 x 3

8. Sharpening vs smoothing In the same way that in the smoothing case, typical spatial techniques for sharpening images are based on kernels. The main difference is the sign of the possible values. In the sharpening case, it will appear negative terms. Most common methods are based on derivatives, such as, Laplacian filter.

9. Sharpening vs smoothing These two operations have an opposite nature , both details or edges of the image and the noise present in the image correspond to high frequencies (for this reason, differentiate them is very difficult) Smoothing intends to remove the high frequencies. Sharpening intends to increase the high frequencies. This is the main problem of the approaches for smoothing and sharpening in a simultaneous way.

10. Sharpening vs smoothing The initial approach is to consider a two-steps process : first smoothing and then sharpening, or the other way around. However, this approaches usually leads too many problems. If we first apply a smoothing technique, then we will be loosing information that cannot be recovered in the succeeding sharpening step. If we first apply a sharpening method over a noisy image, we will amplify the noise presented in it.

11. Sharpening vs smoothing The initial approach is to consider a two-steps process : first smoothing and then sharpening, or the other way around. However, this approaches usually leads too many problems. There are any author that proposes approaches for simultaneous sharpening and smoothing. One of the best approaches is BM3D-SH3D , proposed by Dabov et. al. However this method are based on non-local means Increased computational cost

12. Contents Sharpening vs smoothing Graph based model for color image processing Application of the model to simultaneous sharpening and smoothing Conclusions

13. Graph based model for colour image processing Each pixel is processed by using a 3x3 sliding window of neighbors with center on it. F 1 F 2 F 3 F 4 F 5 F 6 F 7 F 8 F 9 Each pixel is represented by its RGB coordinates:  

14. Graph based model for colour image processing complete weighted graph associated to  

15. complete weighted graph associated to   Graph based model for colour image processing

16. complete weighted graph associated to   An edge exists between two pixels, if is lower than a certain threshold .   Graph based model for colour image processing

17. The value of the threshold is one of the keys of the model. It ensures us that two pixels will be joined if they are similar.   Flat region Graph based model for colour image processing

18. Detail region The value of the threshold is one of the keys of the model. It ensures us that two pixels will be joined if they are similar.   Graph based model for colour image processing

19. The feature that better characterizes whether a pixel belongs to a flat region or a detail region is the cardinal of the nodes set of the connected component that contain the central pixel, that we denote by . We define a border detection based on this cardinal to segment noise free images A grayscale image was created from this cardinal: the bigger cardinal, the greater intensity of greys.   Adjustment of the threshold

20. Adjustment of the threshold We obtain the optimal threshold by maximizing the mutual information between the grayscale image obtained using and the output of a fuzzy edge detector.  

21. We are interested in a robust method in noisy environments, therefore, we add white Gaussian noise with different standard deviations to all the set of images. In analogous way that before, the optimal threshold for the noisy images was calculated maximizing the mutual information . Adjustment of the threshold

22. The standard deviation of the Gaussian noise of an image is unknown in general. A noise variance estimation was calculated for a set of images. Using linear regression analysis , we obtain a relation between this estimation and the optimal threshold, that enables us to fix the threshold without prior information of the image . Adjustment of the threshold

23. Contents Sharpening vs smoothing Graph based model for color image processing Application of the model to simultaneous sharpening and smoothing Conclusions

24. Application of the model to simultaneous Smoothing and Sharpening We have each pixel of the image modelled by a local graph. This graph allow us to know the local structure of the image, information that permit us to apply smoothing and sharpening in a simultaneous way. We are going to use

25. Application of the model to simultaneous Smoothing and Sharpening We have each pixel of the image modelled by a local graph. This graph allow us to know the local structure of the image, information that permit us to apply smoothing and sharpening in a simultaneous way. We are going to use The connected component of the central pixel for apply smoothing .

26. Application of the model to simultaneous Smoothing and Sharpening We have each pixel of the image modelled by a local graph. This graph allow us to know the local structure of the image, information that permit us to apply smoothing and sharpening in a simultaneous way. We are going to use The others components for apply sharpening . The connected component of the central pixel for apply smoothing .

27. Application of the model to simultaneous Smoothing and Sharpening We design a 3 x 3 kernel for each pixel so that  

28. Application of the model to simultaneous Smoothing and Sharpening Where is a parameter for controlling the smoothing effect.   We calculate the denoised value , by as:     We design a 3 x 3 kernel for each pixel so that

29. Where is a parameter for controlling the sharpening effect.   Application of the model to simultaneous Smoothing and Sharpening We design a 3 x 3 kernel for each pixel so that   And finally, we calculate the denoised-sharpened value by as :    

30. Application of the model to simultaneous Smoothing and Sharpening In this way, the final proposed kernel is given by:  

31. Experimental results Noisy image Flat Region: APPLY METHOD Only Smoothing

32. Experimental results Noisy image Border Region: APPLY METHOD Smoothing and Sharpening

33. Experimental results Noisy image Border Region: APPLY METHOD Smoothing and Sharpening

34. Experimental results     Only smoothing Noisy image Proposed method Proposed method applied to a noisy image

35. Contents Sharpening vs smoothing Graph based model for color image processing Application of the model to simultaneous sharpening and smoothing Conclusions

36. Conclusions We have constructed a model based on graph theory able to characterize a color image and differentiate appropriately texture regions and flat regions . Using this model, a new spatial filter for color image smoothing and sharpening have been designed. This filter is able to enhance the details of the image at the same time that the noise is removed.

37. Future Work Optimize the parameters of the model and obtain the better combination between sharpening and smoothing. Measure the performance of the proposed method by any non-reference image quality measure. Join the two parameters in order to have only one parameter in the model that allow us to control the smoothing-sharpening effect.

38. Thanks for your attention !

39. Thanks for your attention !