Image Generation with ComfyUI and Stable Diffusion

Discover the power of local image generation through a gradual use case to generate yoga poses and learn everything about Stable Diffusion, text-to-image, image-to-image, embedding, LoRA and ControlNet!

1. Get Comfy and Dream!
Controlled Image Generation
with Stable Diffusion
Hajer Mabrouk
Raphaël Semeteys

2. •Former Innovation Manager at Oracle
•Certified Somatic Coach and Yoga Teacher
•Head of DevRel, Senior Architect at Worldline
•Certified Yoga Teacher
Hajer Mabrouk Raphaël Semeteys
linkedin.com/in/ hajer-mabrouk/ raphiki.github.io

3. Use Case
•YogĀrkana is a website dedicated to yoga
•Descriptions and images of yoga poses
must be precise
Locally generate accurate images of Yoga Poses
•Photography is not always the best option
•Images or photos from the internet cannot
be reused
How could Generative AI help?

4. Stable Diffusion
From German Labs to London-based Startup
•Collaboration of several companies and German Labs
•Latent Diffusion Model with embedding space in 2021
•CLIP-guided diffusion
•LIAON dataset
•Runway and EleutherAI participation
•Stability AI
•Compute donation to the project
•Hired most of initial researchers
•Now official maintainer of Stable Diffusion models
“Open” Licenses
•Responsible AI: OpenRAIL
•Version 3.5: Enterprises with
1M+ revenue must pay

5. Stable Diffusion
Very dynamic contributing Communities
Models
•Fine-tuning: Custom Models for
specific styles or themes
•Refiners, Upscalers, ControlNets
•Model extensions (LoRA)
Tools
•User-Friendly Interfaces:
Automatic1111 Web UI, ComfyUI
•Fine-tuning tools: Dreambooth,
Kohya SS
Sharing Communities
•Portals to share models, prompts, images and tutorials: Hugging Face, Civit.ai…
•Stable Horde: crowdsourced distributed cluster of generation workers

6. GUI for local Stable Diffusion workflows
•Intuitive, modular and customizable
•Flexible node-based workflows
•Text-to-Image Generation
•Image-to-Image Processing
•Custom Node Management
•Community-Driven
•GPL 3 License
•Contributions, plugins, doc

7. Let’s start with a simple demo
Generate a image of a girl
doing a yoga pose

8. What does Stable Diffusion?
•Starts with Random Noise: Begins with a noisy,
unrecognizable image
•Refines Step-by-Step: Gradually removes noise,
adding details
•Learns from Real Images: Uses patterns from
trained images
•Text-Guided Creation: Follows prompts like "sunset
over mountains"
•Denoising Process: Clarifies image layer by layer
•Final Image Output: Produces a clear, detailed
image matching the prompt
Source: wikipedia

9. Models
•Most used Stability.ai Models
•SD 1.5
•SDXL
•Fine-tuned Models
•Specialized: style, subject
•Shared by communities (like civitai.com)

10. Prompts
•CLIP Model (Contrastive Language–Image Pretraining)
•Connect descriptive text and images
•Help generate images matching specific prompts
•Can handle a wide range of prompts
•Developed by OpenAI in 2021
•Usable under the MIT license
•Trained on 400M image-text pairs from the Internet
•Positive & Negative
•Textual or Short syntax

11. Embeddings (Textual Inversions)
•Vector representations of text
•“Instructions” for image generation
•Style, theme, texture, pose, character features, etc.
•Small files containing additional concepts
•To be injected in prompts
•Community provides many presets
•Must aligned with Stable Diffusion version

12. Embeddings (Textual Inversions)
GhibliFantasyComic3D Render
Analog Film Cinematic Cyberpunk Digital Art
No Embedding
Vector Art

13. Latent Space
•Latent Space
•Abstract, compressed representation of the image
•Handles encoded features such as shapes, colors,
textures and general structure
•Manipulation of embedding vectors
•Iterative and refining generation
•Random noise is introduced into the latent space
•At each step the model adjusts the features to match
the prompt
•VAE (Variational Autoencoder)
•Convert Image Pixels → Latent Space

14. Denoising Process
•Seed
•Random seed used to create initial noise
•Fixing it allows to see impact of other parameters
•Samplers
•Algorithms guiding the iterative image generation
•Differ in Speed and Quality
•Schedulers
•Control how noise is removed at each step
•Also impact Speed and Quality, Karras is well balanced
•Other Parameters
•#steps, CFG (adherence to prompt), %denoising

15. I can tweak generation
but I don’t control the pose…
Camel poseTree poseLotus pose Shoulder Stand pose
Text-to-Image generation is not enough!

16. Let’s move on to Image-to-Image
Generate a image of a girl
doing a yoga pose based on
an existing image

17. Image-to-Image Generation
CFG 20 CFG 8
•Input Image
•Replace the Empty Latent Image with a real one
•Need a VAE Encode (from the model)
•Play with % denoising
•Prompt has less impact
•Increasing CFG only reduces quality
denoise 0.55 denoise 0.70

18. ControlNets
•Specialized Neural Networks
•Additional control and guidance to primary model
•Use reference images to transfer structural information
or inject features
→ Hybrid approach with both text and visual references
•Control methods
•Structural: pose, edge detection, segmentation, depth
•Texture & Detail: scribble/sketch, stylization from edges
•Content & Layout: bounding boxes, inpainting masks
•Abstract & Style: color maps, textural fields
Depth ControlNet

19. Preprocessors for ControlNets
Initial Image Line Art Color Map Open Pose
Segmentation Depth Map ScribbleStraight Lines

20. More abstract input images
•Design poses in 3D with image export
•Use of JustSketchMe tool (webapp & PWA)
•Design poses based on my own knowledge
•Several angles of view
•(waiting for 3D GenAI Models)

21. How can I achieve greater
consistency for the character?
Create images featuring the
same facial identity

22. LoRA
•Low-Rank Adaptation
•Lightweight Model Adaptation
•Update a small subset of model parameters
•Very efficient
•Small File Size, use significantly less memory
•Faster Training
•Usage
•Specific styles, poses, characters, or concepts
•Triggered by keywords in the prompt
•Many LoRAs are provided by the community

23. Examples of generated images

24. Controlled generation for Diversity & Inclusion

25. Controlled generation for Diversity & Inclusion

26. Conclusion
•Image Generation is both Science and Art
•A lot of parameters to tune
•Additional inputs and components to control generation
•Our use case is implementable
•Nicer and homogeneous images for YogĀrkana
•Cheery on the cake: a more inclusive Website!
•Next steps
•Create our own LoRA, test video generation
•Explore voice generation for i18n and more inclusivity
From
to
&

27. Namaste!
raphiki.github.io
deck.yogarkana.com