240520_Thanh_LabSeminar[G-MSM: Unsupervised Multi-Shape Matching with Graph-based Affinity Priors].pptx

G-MSM: Unsupervised Multi-Shape Matching with Graph-based Affinity Priors

1. G-MSM: Unsupervised Multi-Shape Matching with Graph-based Affinity Priors Tien-Bach-Thanh Do Network Science Lab Dept. of Artificial Intelligence The Catholic University of Korea E-mail: os fa19730 @catholic.ac.kr 202 4/05/20 Marvin Eisenberger et al. CVPR 2023

2. Introduction Shape matching of non-rigid object categories is a central problem in 3D CV The majority of existing DL methods for shape matching treat a given set of meshes as an unstructured collection of poses Random pairs of shapes are sampled for NN and pairwise matching loss is minimized Fail to recognize commonalities and context-dependent patterns Not all samples of a shape collection are created equal, some pairs of poses are much closer than others Define an affinity graph, undirected shape graph G on the set of input shapes whose edge weights (affinity scores) are informed by the outputs of pairwise matching module Define multi-matching architecture that propagates matches along shortest paths in the underlying shape graph G, apply cycle-consistency for optimal paths in the G

3. Introduction

4. Related Works Axiomatic correspondence methods Learning-based methods Multi-shape matching

5. Method Problem formulation Consider a collection of 3D shapes S = {X(1),…, X(N)} from non-rigidly deformable shape categories Each shape is a discretized approximation of 2D Riemannian manifold X(i) = (V(i), T(i)) where V is set of nodes and T is triangular faces Goal is construct an algorithm computes dense correspondence mapping between any two surfaces from the shape collection Demonstrate proposed multi-matching approach excels, including non-isometric pairs, poses with topological noise from self-intersections, and inter-class matching

6. Method Network architecture

7. Method Network architecture Consist of 3 separate components, first 2 modules are standard components namely learable feature backbone I and a pairwise matching layer II, finally the multi-matching architecture III Feature extractor defined as given input shape X(i) = (V(i), T(i)), the mapping produces an l-dimensional feature embedding F per node V ( DiffusionNet ) Pairwise matching is multi-scale matching scheme based on DeepShells Given transport plan, the energy specifies the distance between the discrete measures associated with 2 arbitrary l-dimensional feature embeddings F and G Take the minimum overall possible transport plans results in Kantorovich formulation of optimal transport Phi match is defined as a deterministic, differentiable function that takes local feature encodings F as input and predicts a set of correspondences II These coordinates specify a registered version of the first input shape that closely aligns with the pose of second input shape. l>0 is a training loss signal

8. Method Network architecture Graph-based multi-shape matching Shape graph: construct G as a complete graph (undirected, fully connected), where missing edge between X(i) and X(j) can be specified equivalently by setting edge weight Define pairwise edge weights represent affinity scores between pairs of shapes Propose heuristic for a given pair of shapes and define symmetric affinity score w as small matching energy implies a high geometric similarity between the input poses Multi-matching Obtain multi-shape matches

9. Method Network architecture Graph-based multi-shape matching Pass along shortest paths in the graph, the approach thereby favors edges with close affinity (small pairwise matching cost) Promote cycle-consistency during training via the loss This loss imposes a penalty on inconsistencies between the registration V produced by the pairwise matching module, and the multi-shape correspondences Overall loss function

10. Experimental Results Dataset FAUST contains 10 humans in 10 different poses each SCAPE contains 71 diverse poses of the same individual SURREAL consists of synthetic SMPL meshes fitted to raw 3D motion capture data SHREC’19 Connectivity contains human shapes in different poses

11. Experimental Results Graph topologies Node is 3D shape Edge set = N(N-1)/2 Minimum spanning tree Travelling salesman problem Star graphs where all nodes are connected to one center node

12. Experimental Results

13. Experimental Results

14. Experimental Results

15. Experimental Results

16. Experimental Results

17. Experimental Results

18. Conclusion G-MSM - a novel multi-matching approach for non-rigid shape correspondence Given collection of 3D meshes, define a shape graph G approximates the underlying shape data manifold Edge weights are extracted from putative pairwise correspondence signals in self-supervised manner Cycle-consistency of optimal paths, produces context-aware multi-matches informed by commonalities and salient geometric features across all training poses