[NS][Lab_Seminar_240826]Learning Graph Neural Networks for Image Style Transfer.pptx

Learning Graph Neural Networks for Image Style Transfer

1. Learning Graph Neural Networks for Image Style Transfer Tien-Bach-Thanh Do Network Science Lab Dept. of Artificial Intelligence The Catholic University of Korea E-mail: os fa19730 @catholic.ac.kr 202 4/08/26 Yongcheng Jing et al. ECCV 2022

2. Introduction What is Image style transfer? Transferring the artistic style from a source image to a content image Applications: digital art, photo editing, visual effects Challenges: Parametric methods: tend to distort local style patterns Non-parametric methods: often introduce artifacts due to patch mismatching

3. Motivation Problems with existing methods: Parametric: good global style transfer but poor local detail preservation Non-parametric: good local detail but prone to mismatches and artifacts Solution: Semi-parametric approach using GNNs to balance global and local style transfer Fig. 1. Existing parametric [14, 1, 30] and non-parametric [6, 42] NST methods either barely transfer the global style appearance to the target [6], or produce distorted local style patterns [14, 1, 30] and undesired artifacts [42]. By contrast, the proposed GNN-based semi-parametric approach achieves superior stylization performance in the transfers of both global stroke arrangement and local fine-grained patterns.

4. Method Overview Semi-parametric approach: Combine the strengths of both parametric and non-parametric methods GNNs are used to establish fine-grained content-style correspondences Key techniques: Graph construction: content and style patches as nodes Message passing: Heterogeneous GNN for style-content interaction Deformable graph convolution: handles cross-scale style-content matching

5. Method Stylization Graph Construction Nodes: content and style patches Edges: Inter-Domain edges: between content and style nodes Intra-Domain edges: within content nodes to ensure style coherence k-NN search Employ distance metric of normalized cross-correlation for pair-wise k-NN Target k-NN style nodes for content patch content and style patches k largest elements from set pair-wise NCC

6. Method Deformable Graph Convolution Need for scale adaptation: different content regions require different style scales (stroke sizes) Solution: deformable scale predictor adjusts patch sizes for optimal style-content matching Style-to-Content Message Passing Apply attention coefficient for each style node during message passing Content-to-Content Message Passing Use Feat2Patch

7. Method Local Patch-based manipulation Patch extraction: content and style feature patches extracted using sliding windows GNNs: Used to model the stylization process as message passing between content and style nodes Adaptive feature aggregation based on attention mechanisms

8. Method Global Feature Refinement Problem: local patch-based manipulation may lack global style coherence Solution: A global feature refinement stage using Adaptive Instance Normalization Ensure the final image preserves both fine details and global style appearance

9. Method Feature Encoding Decoding process: convert the refined features back into image Components: convolutional and bilinear upsampling layers with ReLU activations Outcome : generate the final stylized image with preserved content and applied style

10. Method Loss Function and Training Strategy Content loss Style loss mapping BN statistics over the feature maps mean standard deviation

11. Method Loss Function and Training Strategy

12. Experiments Experimental Settings Stylization graph construction stage, k = 5 by default for NCC-based KNN search Sliding window = 1, kernel size = 5*5 Employ pre-trained VGG-19 Dataset Microsoft COCO WikiArt Deep Graph Library

13. Experiments Qualitative comparison Fig. 3. Qualitative results of our proposed GNN-based semi-parametric stylization algorithm and other parametric [30, 14, 1] and non-parametric [6, 42] methods.

14. Experiments Efficiency analysis Table 1. Average speed comparison in terms of seconds per image

15. Experiments Ablation Studies - Heterogeneous aggregation schemes Fig. 4. Comparative results of using various aggregation mechanisms for heterogeneous message passing, including graph attention network (GAT) [43], graph convolutional network (GCN) [24], graph isomorphism network (GIN) [48], dynamic graph convolution (EdgeConv) [46], and GraphSage [12]. The GAT mechanism generally yields superior stylization results, thanks to its attention-based aggregation scheme in Eq. 2.

16. Experiments Ablation Studies - Stylization w/ and w/o the deformable scheme Fig. 5. Results of the equal-size patch division method and the proposed deformable one with a learnable scale predictor. Our deformable scheme allows for cross-scale style content matching, thereby leading to spatially-adaptive multi-stroke stylization with an enhanced semantic saliency (e.g., the foreground regions of the horse and squirrel)

17. Experiments Ablation Studies - Graph w/ and w/o intra-domain edges Fig. 6. Results of removing the content-to-content intra-domain edges (w/o Intra) and those with the intra-domain ones (w/ Intra). The devised intra-domain connections incorporate the inter-relationship between the stylized patches at different locations, thereby maintaining the global stylization coherence (e.g., the eye regions in the figure)

18. Experiments Ablation Studies - Euclidean distance vs normalized cross-correlations Fig. 7. Results obtained using Euclidean distance and normalized cross-correlation (NCC) for similarity measurement during the construction of heterogeneous edges

19. Experiments Ablation Studies - Various patch sizes Fig. 8. Results obtained using various patch sizes for constructing content and style vertices in the local patch-based manipulation module. By using a larger patch size, the stylized results can maintain an overall larger stroke size

20. Experiments Diversified Stylization Control Fig. 9. Flexible control of diversified patch-based arbitrary style transfer during inference. The proposed GNN-based semi-parametric stylization scheme makes it possible to generate heterogeneous style patterns with only a single trained model.

21. Conclusion Key contributions: Introduce a semiparametric style transfer method using GNNs Achieve superior results in both global and local style transfer Future work: Extend the GNN-based approach to other vision tasks Optimize the k-NN search process for faster stylization