Build applications with generative AI on Google Cloud

We will explore Vertex AI - Model Garden powered experiences, we are going to learn more about the integration of these generative AI APIs. We are going to see in action what the Gemini family of generative models are for developers to build and deploy AI-driven applications. Vertex AI includes a suite of foundation models, these are referred to as the PaLM and Gemini family of generative ai models, and they come in different versions. We are going to cover how to use via API to: - execute prompts in text and chat - cover multimodal use cases with image prompts. - finetune and distill to improve knowledge domains - run function calls with foundation models to optimize them for specific tasks. At the end of the session, developers will understand how to innovate with generative AI and develop apps using the generative ai industry trends.

1. Build applications with generative AI
on Google Cloud
Márton Kodok
Software Architect at REEA.net
June 13-14, 2024 Barcelona

2. 1.Intro
2.Build with Gemini
3.What is Vertex AI?
4.Foundation Models + Model Garden
5.Multimodal use cases + Demo
6.Function calling with Gemini + Demo
Agenda
Build applications with generative AI on Google Cloud @martonkodok

3. ●Google Developer Expert on Cloud technologies (2016→)
●Champion of Google Cloud Innovators program (2021→)
●Among the Top 3 romanians on Stackoverflow 206k reputation
●Crafting Cloud Architecture+ML backends at REEA.net

Articles: martonkodok.medium.com
Twitter: @martonkodok
Slideshare: martonkodok

StackOverflow: pentium10
GitHub: pentium10

Build applications with generative AI on Google Cloud @martonkodok
About me

4. “
@martonkodok

5. “
demo…
@martonkodok

6. @martonkodok
Gemini Extensions

7. @martonkodok
Getting results from Gmail

8. @martonkodok
list the top 10 incidents from email for service name "trademark", group results in table

9. @martonkodok
Sum my invoices

10. @martonkodok
find all emails from this year from Hidroelectrica that contain "factura" and list all invoice values
for this year, with dates, in a table, order by descending

11. @martonkodok01
Build with
Gemini
Part #1

12. @martonkodok
Gemini comes in different sizes
Gemini 1.0 Pro
Multimodal reasoning across a
wide range of tasks
GA
Gemini 1.5 Flash
fastest, most cost-efficient
model yet for high volume tasks
GA
Gemini 1.5 Pro
Multimodal reasoning for longer
prompts, 1 million context
window
GA
1M

high volume 1000 (RPM),
lower latency and cost
1M

Large prompt,
1000+ pages of PDF
32K

Text, 16 images,
1 video

13. * May 2024 @martonkodok
A bit about pricing
For 1M tokens Gemini 1.5 FlashLlama 3 70B C3 Haiku GPT-4o
Input Token $0.35 - $0.70 $0.59 $0.25 $5.00
Output Token $1.05 - $2.10 $0.79 $1.25 $15.00
1 million tokens (for prompts up to 128K tokens | for prompts longer than 128K)

14. @martonkodok
Where to use
Build with Gemini
Google AI Studio
aistudio.google.com
- API key
- 60 QPM - free outside of EU
- Libraries in Python/Go/Node/Dart/Android
console.cloud.google.com
- service account key
- Vertex AI SDK
- LangChain integration + libraries
- access to 120 models via Model Garden
- available for individual/edu/business
Vertex AI Studio

15. “
Vertex AI is the Google Cloud product group for ML
End-to-end ML Platform
@martonkodok

16. Gen AI and Model Garden on Vertex AI @martonkodok
Google Cloud AI Portfolio
Foundation
Models
Vertex AI
End-to-End ML Platform
Text Chat Code ImageVideo
Google Cloud Infrastructure - GPUs/TPUs
Contact Center AI Healthcare AIDiscovery AIDocument AI
Business Users
AI Practitioners
Developers
Audio and
Music
Vertex AI
Studio
Generative AI
APIs
Model Garden
Gemini Business
Google Workspace
Duet Al for
Google Cloud
AI Platform: Fine tuning | Distill | Serve

17. 02
Vertex AI
+
Model Garden
Part #2

18. Build applications with generative AI on Google Cloud @martonkodok
Model Garden
Task Specific
AutoML and APIs
Open Source
Models
Foundation
Models
Model Garden

19. Build applications with generative AI on Google Cloud @martonkodok
Model Garden
Foundation Models
Multi-task Large-scale Minimal training

20. Build applications with generative AI on Google Cloud @martonkodok
Language Foundation Models
Gemini 1.0 Pro
Multimodal reasoning across a
wide range of tasks
Gemini 1.5 Flash
fastest, most cost-efficient
model yet for high volume tasks
Gemini 1.5 Pro
Multimodal reasoning for longer
prompts, 1 million context
window
Imagen 2
Generate images from
Text prompts

Multimodal
Embeddings
Extract semantic information


Chirp for
Speech to Text
Build voice enabled applications

21. Build applications with generative AI on Google Cloud @martonkodok
Introduction
Generative AI Studio
Console-tool Test generative models Rapid prototyping

22. Build applications with generative AI on Google Cloud @martonkodok
Generative AI Studio

23. Build applications with generative AI on Google Cloud @martonkodok
Custom language Prompt samples
1.Summarization
2.Classification
3.Extraction
4.Writing
5.Ideation
cloud.google.com/vertex-ai/docs/generative-ai/learn/prompt-samples

24. “
Vertex AI
console demo
@martonkodok

25. Build applications with generative AI on Google Cloud @martonkodok
Prompt structure
1.Text:
2.Question:
3.Answer:
4.Categories:
5.Options:

26. Build applications with generative AI on Google Cloud @martonkodok
Best practices for the context field of your prompt

27. Part #3
03
Gemini
Multimodal
in action

28. Build applications with generative AI on Google Cloud @martonkodok
Introduction

Multi model
Image Video Text Function Calling

29. Build applications with generative AI on Google Cloud @martonkodok
Multi model use cases
1.Info seeking
2.Object recognition
3.Digital content understanding
4.Structured content generation
5.Captioning / description
6.Extrapolation

30. Build applications with generative AI on Google Cloud @martonkodok
Multi model use cases

31. “
demo…
@martonkodok

32. Build applications with generative AI on Google Cloud @martonkodok
Multimodal - Embeddings
Multi model - Embeddings
Image Video Text Audio

33. Build applications with generative AI on Google Cloud @martonkodok
Language Foundation Models
Gemini 1.0 Pro
Multimodal reasoning across a
wide range of tasks
Gemini 1.5 Flash
fastest, most cost-efficient
model yet for high volume tasks
Gemini 1.5 Pro
Multimodal reasoning for longer
prompts, 1 million context
window
Imagen 2
Generate images from
Text prompts

Multimodal
Embeddings
Extract semantic information


Chirp for
Speech to Text
Build voice enabled applications

34. Google Cloud Next ‘24 Proprietary
Joint Embedding
Vector Space

Image:
“gray tabby cat
laying in front of a
Christmas tree”
Text:
size color
living
Multimodal
embeddings
●Embed text, image, and video in the
same semantic space with the same
dimensionality
●Applications: image/video content
search, classification,
recommendation

35. 035Google Cloud Next ‘24 Proprietary
Resources

BQ Embedding Generation
Documentation | Blog | Video

BQ Vector Search

Documentation | Blog | LangChain

Demo Assets

BQ LangChain Notebook |
Gen AI Repo
Vector search and multimodal
embeddings in BigQuery
goo.gle/next24-ana302

36. “
Cloud Next’24 demo app in action
@martonkodok

37. 04
Gemini
Function Calling
Part #4

38. Build applications with generative AI on Google Cloud @martonkodok
Function calling in Gemini
A framework to connect LLMs
to real-time data

delegate certain data processing tasks
to functions

39. cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling @martonkodok
Function calling in Gemini

40. Gen AI … repository

Build applications with generative AI on Google Cloud @martonkodok

41. @martonkodok
Sample code for GenAI on Google Cloud
goo.gle/gen-ai-github

42. Build applications with generative AI on Google Cloud @martonkodok
https://goo.gle/gen-ai-github -> Gemini folder

43. “
Function calling demo…
@martonkodok

44. Part #5
05
Open-source
models

45. Build applications with generative AI on Google Cloud @martonkodok
Open Source models
Llama 2 on Vertex AI
Meta's Llama 2 models
on Vertex AI
Code Llama
Designed for general code synthesis and
understanding, designed for Python
Gemma 2B and 7B
Family of lightweight,
state-of-the-art open models
Claude on Vertex AI
Claude 2, Instant 1.2, and more

46. “

Vertex AI enables model inference via API
Build applications with generative AI on Google Cloud @martonkodok

47. Build applications with generative AI on Google Cloud @martonkodok
VertexAI: Managed unified ML platform
Fine-tuning
1 click deploy

48. @martonkodok
Explore AI models with Colab samples
cloud.google.com/vertex-ai/docs/start/explore-models#test-promptsub

49. Conclusion

Build applications with generative AI on Google Cloud @martonkodok

50. 1.Build with the groundbreaking ML tools that power Google
2.Model Garden provides a curated collection of 120+ models
3.Approachable from the non-ML developer perspective (managed models, fine-tuning training)
4.Accelerate ML with tooling for pre-trained, open source and custom models
5.Deploy to applications with just one-click
6.End-to-end integration for data and AI with build pipelines that outperform and solve complex ML tasks
Vertex AI: Enhanced ML developer experience
Build applications with generative AI on Google Cloud @martonkodok

51. Twitter: @martonkodok




Thank you. Q&A.



Reea.net - Integrated web solutions driven by creativity
to deliver projects.

Follow for articles:
martonkodok.medium.com
Slides available on:
slideshare.net/martonkodok