SFSCON23 - Markus Pobitzer - Image Generation with Diffusion Models

Recent machine learning developments saw a breakthrough in generating images. So-called Diffusion Models can create photo-realistic images from noise. With the help of an input text (prompt) we can guide the generation and produce matching images.

This technology opened new doors for creating digital art, modifying existing images, and creating stunning visual experiences. In the talk, we will find out how these algorithms work, introduce Stable Diffusion (a concrete implementation), and find out what its use cases are. We will see how text can be used to generate matching outputs but also take a look at more experimental features such as creating images from edges, outlines, or depth maps.

We will mainly focus on the open source text-to-image model Stable Diffusion, which has set new standards in image generation. With it also comes an active community that keeps it open source and accessible for everyone.

1. “Standing on top of the highest mountain, looking down to the other peaks, alps.
An award-winning landscape photo of South Tyrol”

2. Image Generation with
Diffusion Models
How computers imagine our world
Markus Pobitzer

3. Overview
Diffusion Models

4. Stable Diffusion (SD)
(Text) Guidance Stable Diffusion Output
“Majestic royal ship on
a calm sea, oil
painting, …”

5. Inpainting with Stable Diffusion
Input Mask Output
Original imagefrom: https://unsplash.com/@overture_creations

6. Billions
ofImage-Text pairs
256
Nvidia A100 (40GB) GPUs
150000
GPU Hours
Training Stable Diffusion(v1)
Source: SD ModelCard, https://github.com/CompVis/stable-diffusion/blob/main/Stable_Diffusion_v1_Model_Card.md

7. •DALL-E
•Imagen
•Midjourney
•StableDiffusion
Image Generation Applications
unreleased
or
behindAPI
trainingcode
and
modelweights
available

8. Beyond Text Guidance
Input (Edges) Outputs

9. “Sign with SFSCON written on it” Real or Fake?
ChallengesShortcomings

11. Tryitout!
Generate theimageon thelefton your
devicethankstoGoogle Colab.
Scan theQR-Code orvisit:
github.com/Markus-Pobitzer/SFSCON-2023