JanusCon 2024: Mom there are robots in my meeting

Slides from my presentation at JanusCon 2024

1. Mon, there are robots
in my meeting!
Saúl Ibarra Corretgé


2. Once upon a time…

7. Set of Open Source projects
Easily deploy scalable and secure video
conferencing solutions.
APIs and mobile SDKs
Integrate Jitsi into existing products to add video
conferencing capabilities.
Community
Open Source enthusiasts from all around the world
contribute to Jitsi.

9. JaaS
Hosted Jitsi, by 8x8 — jaas.8x8.vc

11. 2 API levels
•High level: iframe based, very
versatile and customisable
•Low level: your own UI

12. Product + Platform
•Work is spread across both
•Improvements in the in-
meeting experience
•Improvements in APIs for
embedders

13. The AI gold rush
AI + meetings = ?

14. Alright, let’s build a chatbot!
•It needs to join the meeting as
a participant
•Interact via text (built-in chat)
•Private messages for questions
•Hot word detection?
•Audio output?
•Video output?
How hard can it be?

15. Demos
Recorded, no sweat !

17. What was that?
•Node app running Chrome via Playwright
•Bot written with lib-jitsi-meet, our low level API
•Google for transcriptions
•ChatGPT for AI
•Zapier for emails

19. What was that?
•Node app running Chrome via Playwright
•Bot written with lib-jitsi-meet, our low level API
•Google for transcriptions
•ChatGPT for AI
•PlayHT for playing back audio
•WebGL for the moving robot

21. But we had other robots before…
•Our recorder is based on a Chrome instance running on a container
•PSTN access is accomplished with a server side WebRTC client
•Our robots are invisible though
•So we used Chrome because YOLO

22. Ian Malcolm, Jurassic Park
“Your scientists were so preoccupied with
whether or not they could, they didn't stop to
think if they should.”

23. How do we interact with chatbots?

24. Private text
conversations
(Mostly)

25. AI is personal

26. Let’s focus on the building blocks
•GenAI has text as its lingua
franca
•Whatever the input it gets
converted to text
•The some LLM operates on that
text
•Last, the result is converted to
the desired output

27. Text is King

28. Start by adding core
primitives
Transcriptions + Summaries

30. Real Time
Usable as subtitles

32. A10

36. How does it all fit together?

37. Getting to users
How would they know to turn transcriptions on?

42. Brave Talk
What are our customers doing?

45. How can you built this?
•Use transcription web hooks to
keep a full transcript
•Use transcription iframe events
to locally display the transcript
+ the full transcript to catch up
•Custom LLM (Brave doesn’t use
Skynet) for prompting

47. What we leaned
•Every time we tried to solve a problem with a robot there was a better way
to solve it without it
•In the context of a meeting
•Understanding transcriptions are a type of recording makes sense
•The first demo was poisonous, complexity wise; start simple!
Hopefully this helps others

48. Try it out!
•JaaS: https://jaas.8x8.vc — Get started for free
•meet.jit.si — Free Jitsi Meet instance
•github.com/jitsi — host it yourself

49. Future with AI
•Current generation of AI is very powerful but still ways to go
•A good future would involve small models for dedicated tasks
•Skynet
•Keep iterating
•Image summaries with LLaVA?
•Will we do bots?

50. Thanks
Tudor Avram Răzvan Purdel

51. That’s all folks!
Questions?
github.com/jitsi



" @jitsinews

" @saghul

# [email protected]