GMM Clustering Presentation Slides for Machine Learning Course

GMM Lecture Slides.

1. DTS304TC: Machine Learning Lecture 8: Gaussian Mixture Model (GMM) Dr Kang Dang D-5032, Taicang Campus [email protected] Tel: [phone]

2. Acknowledges This set of lecture notes has been adapted from materials originally provided by Dr. Gan Hong Seng and Christopher M. Bishop's lecture notes. 2

3. Course Outline What it is GMM? The concept of Mixture of Gaussians EM algorithm & Latent Variables l, 3

4. What is Gaussian Mixture Model? Probabilistic Model used for clustering and classification tasks. Assumption: data is generated by a mixture of several Gaussian distributions, each with its own mean and variance. Application: by fitting a GMM to the data: Identify underlying clusters. Make predictions on new data points through probabilistic assignments to each cluster. . What is Gaussian Mixture Model 4

5. Example of Gaussian Distribution X-Axis: Data Values Y-Axis: Frequency or Probability of Occurrence Bell-Shaped Curve: illustrates that most data is clustered around the mean. Mean is depicted by the vertical line at the center. Standard Deviation measures the spread of the data 5

6. Example of Gaussian Distribution 6

7. Multivariate Gaussian Distribution 7

8. Likelihood Function Data set The probability of observing x given the Gaussian distribution: Assume observed data points generated independently This probability is a function of the parameters this is known as the likelihood function  

9. Maximum Likelihood Obtaining the parameters by the given dataset, and maximizing the likelihood function Equivalently maximize the log likelihood

10. Maximum Likelihood Solution Maximizing w.r.t. the mean gives the sample mean Maximizing w.r.t covariance gives the sample covariance

11. Mixture Models So estimating parameters for a single Gaussian is simple. How about modelling non-Gaussian data? Mixture models can be powerful to handle many non-gaussian data distributions! 11

12. Mixture Model Mixture Models are a collection of the weighted sum of a number of probability density functions (PDFs) where the weights are determined by a distribution   12

13. Example of Mixture Model 13

14. Hard Assignments (K-Means Clustering) Exclusive Assignment: each data point is assigned to a single cluster. Cluster Membership: data points belong to one, and only one, cluster. 14

15. Soft Assignments (GMM) Probabilistic Assignment: Assigns a probability for each data point indicating its likelihood of belonging to each Gaussian distribution in the mixture. Partial Membership: A single data point can have partial membership in multiple Gaussian distributions. 15

16. Q&A When to use hard assignment and when to use soft assignment? 16

17. Hard vs Soft Assignemnts When to Use Hard Assignments Ideal for data with clearly separable, distinct clusters. Most effective when there is minimal overlap between clusters. When to Use Soft Assignments Suitable for data that is not easily separable into distinct clusters. Ideal for handling data with significant overlap between clusters. 17

18. Hard Assignments vs Soft Assignments 18

19. Mixture of Gaussian in 1D 19

20. Mixture of Gaussian in 2D Model Assumption: Data points are generated by a combination of several 2D Gaussian distributions. Distinct Parameters: Each distribution has its own mean (center point) and covariance matrix (shape and orientation). 20

21. Parameters of PDF 21

22. What is inside GMM? 22

23. Gaussian Mixture Model as PDF Q&A: How to prove a function is a PDF? 23

24. Gaussian Mixture Model as PDF 24

25. Gaussian Mixture Model as PDF 25

26. Gaussian Mixtures Linear super-position of Gaussians Normalization and positivity require Can interpret the mixing coefficients as prior probabilities

27. Sampling from the Gaussian Mixture To generate a data point: first pick one of the components with probability then draw a sample from that component Repeat these two steps for each new data point

28. Fitting the Gaussian Mixture We wish to invert this process – given the data set, find the corresponding parameters: mixing coefficients means covariances If we knew which component generated each data point, the maximum likelihood solution would involve fitting each component to the corresponding cluster Problem: the data set is unlabelled We shall refer to the labels as latent (= hidden) variables

29. Synthetic Data Set Without Labels

30. Posterior Probabilities We can think of the mixing coefficients as prior probabilities for the components For a given value of we can evaluate the corresponding posterior probabilities, called responsibilities These are given from Bayes’ theorem by

31. Posterior Probabilities (colour coded)

32. Posterior Probability Map

33. Maximum Likelihood for the GMM The log likelihood function takes the form Note: sum over components appears inside the log There is no closed form solution for maximum likelihood

34. Problems and Solutions How to maximize the log likelihood solved by expectation-maximization (EM) algorithm This is the topic of our lecture How to avoid singularities in the likelihood function solved by a Bayesian treatment How to choose number K of components also solved by a Bayesian treatment

35. EM Algorithm – Informal Derivation Let us proceed by simply differentiating the log likelihood Setting derivative with respect to equal to zero gives giving which is simply the weighted mean of the data

36. EM Algorithm – Informal Derivation Similarly for the covariances For mixing coefficients use a Lagrange multiplier to give

37. 37 EM Algorithm for GMM Estimation

38. 38 EM Algorithm for GMM Estimation

39. 39 EM Algorithm for GMM Estimation - Summary Evaluate the log likelihood

40. EM Algorithm – Informal Derivation An iterative scheme for solving them: Make initial guesses for the parameters Alternate between the following two stages: E-step: evaluate responsibilities 2. M-step: update parameters using ML results

41. Christopher M. Bishop

42. 47 GMM and K-Means Differences K-means Clustering Assumption: Spherical clusters with equal probability. Cluster Assignment: Hard assignment (points belong to one cluster). Cluster Shape: Only identifies circular clusters. Algorithm: Minimizes within-cluster variance. Outlier Sensitivity: High, due to mean calculation. Gaussian Mixture Models (GMM) Assumption: Data from multiple Gaussian distributions. Cluster Assignment: Soft assignment (probabilistic cluster membership). Cluster Shape: Identifies elliptical clusters. Algorithm: Maximizes likelihood using expectation-maximization. Outlier Sensitivity: Lower, due to probabilistic framework.

43. 48 GMM and K-Means Differences Flexibility in Cluster Shapes: GMM can model elliptical and varying size clusters, not just spherical. Soft Clustering and Uncertainty: Provides membership probabilities, offering a nuanced understanding of cluster belonging. Density Estimation: GMM estimates the density distribution of each cluster, not just central tendency. Model Complexity: GMM captures complex cluster structures but requires more data and computational power.

44. 49 GMM and K-Means Differences Use K-means When: You need a fast, simple, and interpretable model. Your data is expected to form spherical clusters. Computational resources are limited. Use GMM When: You suspect clusters are non-spherical or have different sizes. You need a measure of uncertainty in cluster assignments. You have enough data to estimate the additional parameters reliably. Takeaway: K-means is efficient for well-separated, spherical clusters. GMM is more flexible, capturing complex cluster shapes and providing probabilistic cluster assignments.