cs236_lecture1_2023.pptx about machine learning

Cs 236 lecture 1

1. CS 236: Deep Generative Models Stefano Ermon Stanford University URL: deepgenerativemodels.github.io

2. Introduction Computer Vision Natural Language Processing Robotics Computational Speech 2 Challenge: understand complex, unstructured inputs

3. Introduction Richard Feynman: “ What I cannot create, I do not understand ” Generative modeling: “ What I understand, I can create ” 3

4. How to generate natural images with a computer? Many of our models will have similar structure (generation + inference) Generation (graphics) Inference ( vision as inverse graphics) Generative Modeling: Computer Graphics 4 Cube(color= blue , position =( x,y,z ), size =…) Cylinder(color= red , position =( x’,y’,z ’), size =..) High level description Raw sensory outputs

5. Statistical generative models are learned from data Priors are always necessary, but there is a spectrum Prior Knowledge (e.g., physics, materials, ..) + Data (e.g., images of bedrooms) Data Prior Knowledge Graphics This course Statistical Generative Models 5 …

6. A statistical generative model is a probability distribution p(x) Data: samples (e.g., images of bedrooms) Prior knowledge: parametric form (e.g., Gaussian?), loss function (e.g., maximum likelihood?), optimization algorithm, etc. It is generative because sampling from p(x) generates new images Statistical Generative Models 6 scalar probability p(x) A probability distribution p(x) Image x …

7. Building a simulator for the data generating process Data simulator Data

8. Building a simulator for the data generating process Data simulator New datapoints Control signals

9. Building a simulator for the data generating process Data simulator Potential datapoints Probability values New datapoints Control signals

10. Building a simulator for the data generating process Data simulator =Statistical model Potential datapoints Probability values New datapoints Control signals

11. Building a simulator for the data generating process Data simulator =Statistical model =Generative model Potential datapoints Probability values New datapoints Control signals

12. Building a simulator for the data generating process Generative model Potential datapoints Probability values New datapoints Control signals

13. Data generation in the real world Generate Stroke paintings to realistic images [ Meng , He, Song, et al., ICLR 2022] “Ace of Pentacles” Language-guided artwork creation https://chainbreakers.kath.io @RiversHaveWings Generate Generative model of realistic images Generative model of paintings

14. Solving inverse problems with generative models Medical image reconstruction [Song et al., ICLR 2022] Generate Generative model of medical images

15. Outlier detection with generative models Outlier detection [Song et al., ICLR 2018] Generative model of traffic signs High probability Low probability

16. Discriminative vs. generative The image X is given . Goal : decision boundary, via conditional distribution over label Y Ex: logistic regression, convolutional net, etc. Decision boundary 16 Discriminative : classify bedroom vs. dining room Generative : generate X The input X is not given. Requires a model of the joint distribution over both X and Y P(Y = Bedroom | X= ) = 0.0001 P(Y = Bedroom , X= ) = 0.0002 … … Y=B , X= Y=B , X= Y=D , X= Y=D , X= Y=B , X= Y=D , X=

17. Discriminative vs. generative Therefore it cannot handle missing data P(Y = Bedroom | X = ) Joint and conditional are related via Bayes Rule : 17 P(Y = Bedroom | X= ) = P(Y = Bedroom, X = ) P( X = ) Discriminative : Y is simple; X is always given, so not need to model P(X = )

18. Images and Text P(image | caption)

19. Class conditional generative models are also possible: It’s often useful to condition on rich side information Y A discriminative model is a very simple conditional generative model of Y: Conditional Generative Models P(X= | Y = Bedroom) P(X= | Caption = “A black table with 6 chairs”) 19 P(Y = Bedroom | X = )

20. Progress in Generative Models of Images -- GANs Ian Goodfellow , 2019 20

21. Progress in Generative Models of Images – Diffusion Models Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, 2021 21

22. Text2Image Diffusion Models 22 User input : An astronaut riding a horse

23. Text2Image Diffusion Models 23 User input : A perfect Italian meal

24. Text2Image Diffusion Models 24 User input :

25. Dalle3 A minimap diorama of a cafe adorned with indoor plants. Wooden beams crisscross above, and a cold brew station stands out with tiny bottles and glasses

26. Menon et al, 2020 Liu al, 2018 Antic, 2020 Progress in Inverse Problems P(high resolution | low resolution) P(full image| mask) P(color image| greyscale)

27. Progress in Inverse Problems User input :

28. Progress in Inverse Problems

29. Progress in Inverse Problems Kawar et al., 2023

30. Medical image reconstruction Forward model is given by physical simulation Sparse-view sinogram Cross-sectional image Sparse-view computed tomography (CT)

31. WaveNet Generative model of speech signals van den Oord et al, 2016c Parametric Concatenative WaveNet Unconditional Text to Speech Music 31

32. Diffusion Text2Speech Generative model of speech signals Betker , Better speech synthesis through scaling 2023 32

33. Audio Super Resolution Conditional generative model P(high-res signal | low-res audio signal) Low res signal High res audio signal Kuleshov et al., 2017 33

34. Language Generation Radford et al., 2019 Demo from talktotransformer.com P(next word | previous words)

35. Language Generation -- ChatGPT To get an A+ in CS236 (Deep Generative Models) at Stanford, you will need to excel in both your understanding of the material and your performance in assignments and exams. Here are some general tips to help you achieve this: 1. **Attend Lectures and Engage Actively**: Attend all lectures and actively engage with the material. Take thorough notes, ask questions, and participate in discussions. This will help you understand the concepts better . 2 . **Read the Assigned Material**: Make sure to read the assigned textbooks, papers, and supplementary materials. Understanding the theoretical foundations is crucial . 3 . **Stay Organized**: Keep a well-organized notebook or digital notes. This will help you quickly review and understand the material. 4. **Seek Help When Needed**: Don't hesitate to ask questions if you're having trouble with a concept. You can ask the professor, teaching assistants, or classmates for clarification. 5. **Complete Assignments Thoroughly**: Take your time to complete assignments, ensuring you fully understand the requirements and concepts involved. Start early and seek help if you're stuck. High-quality assignments are often a significant portion of your grade. 6. **Collaborate, but Don't Plagiarize**: Collaboration is often encouraged, but make sure you understand your institution's policy on collaboration and plagiarism. Always give credit where it's due and submit original work. … 15. **Meet with the Professor**: If you're aiming for an A+, consider scheduling meetings with the professor to discuss your progress and seek feedback. Remember that getting an A+ can be highly competitive, and the specific grading criteria may vary from one course to another and one professor to another. It's essential to understand the grading policies and expectations of your instructor. Always aim for excellence, but also keep in mind that learning and understanding the material should be your primary goal. Good luck!

36. Machine Translation Conditional generative model P( English text| Chinese text) 36 Figure from Google AI research blog.

37. Code Generation OpenAI Codex

38. Video Generation Suddenly, the walls of the embankment broke and there was a huge flood

39. Video Generation a couple sledding down a snowy hill on a tire roman chariot style

40. Video Generation

41. Imitation Learning Li et al., 2017 41 Conditional generative model P(actions | past observations) Janner et al., 2022

42. Molecule generation

43. DeepFakes Which image is real?

44. DeepFakes

45. Roadmap and Key Challenges Representation : how do we model the joint distribution of many random variables? Need compact representation Learning : what is the right way to compare probability distributions? Inference : how do we invert the generation process (e.g., vision as inverse graphics)? Unsupervised learning: recover high-level descriptions (features) from raw data 45

46. Syllabus Fully observed likelihood-based models Autoregressive Flow-based models Latent variable models Variational learning Inference amortization Variational autoencoder Implicit generative models Two sample tests, embeddings , F-divergences Generative Adversarial Networks Energy Based Models Score-based Diffusion Generative Models Learn about algorithms, theory & applications