How can AI be used in Content Management Systems, Plone Conference 2025

This talk will show multiple examples and demos of AI in action using Volto and Nick.

1. HOW CAN AI BE USED
IN CONTENT MANAGEMENT SYSTEMS?
Plone Conference 2025, Jyväskylä
- Rob Gietema@robgietema

2. WHAT IS AI?
Machine Learning (ML)
Large Language Models (LLM)
...

3. NEURAL NETWORK

4. TYPES OF LLM'S
Generic
Image / Video generation
Image Recoginition
Embedding

5. GENERIC LLM'S
ChatGPT
Claude
Gemini
Deepseek
Qwen

6. LOCAL LLM'S?

7. WHAT DO I NEED TO RUN AN LLM
LOCALLY?
Model
Runner

8. MODAL: QWEN

10. RUNNER: OLLAMA

11. RUNNING LOCAL USING OLLAMA
$ ollama pull qwen3
http://localhost:11434/api/chat

12. HOW TO USE AN LLM IN A CMS?
We need a backend which proxies the requests

13. MEET NICK
Headless CMS
Build with Node.js & PostgreSQL
RESTfull API compatible with the Plone Restapi
Has support for AI models

14. CONFIG IN NICK
export const config = {
...
profiles: [
`${__dirname}/src/profiles/core`,
`${__dirname}/src/profiles/ai`,
`${__dirname}/src/profiles/default`,
],
...
ai: {
models: {
llm: {
name: 'qwen3',
api: 'http://localhost:11434/api/chat',
contextSize: 10,
enabled:true,

15. ENDPOINT IN NICK
POST /@chat HTTP/1.1
Accept: application/json
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ
Content-Type: application/json
{
"prompt": "What is the largest land animal?",
"context": [ ... ],
"messages": [ ... ],
}

16. RESPONSE
HTTP/1.1 200 OK
Content-Type: application/json
{
"model": "qwen3",
"created_at": "2025-01-01T00:00:00.00000Z",
"response": "The largest land animal is the African bush ele
"done": true,
"done_reason": "stop",
"context": [
...
]
"total_duration": 356186167,
"load_duration": 18592125,
"promptevalcount":139,

17. STREAMING
POST /@chat HTTP/1.1
Accept: application/json
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ
Content-Type: application/json
{
"prompt": "What is the largest land animal?",
"context": [ ... ],
"messages": [ ... ],
"stream": true,
}

18. ASSISTANT

19. CHAT HISTORY
POST /@chat HTTP/1.1
Accept: application/json
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ
Content-Type: application/json
{
"prompt": "Tell me a bit about them",
"context": [ ... ],
"messages": [
{
"role": "user",
"content": "What is the largest land animal?"
},
{
"role":"assistant",

20. HISTORY

21. SPEECHRECOGNITION
// Initialize SpeechRecognition
const SpeechRecognition = window.SpeechRecognition ||
window.webkitSpeechRecognition;
recognitionRef.current = new SpeechRecognition();
recognitionRef.current.continuous = true;
recognitionRef.current.interimResults = true;
// Attach event listeners
recognitionRef.current.onstart = handleStart;
recognitionRef.current.onend = handleEnd;
recognitionRef.current.onerror = handleError;
recognitionRef.current.onresult = handleResult;

22. SPEECHRECOGNITION

23. IMAGE / VIDEO GENERATION
Generate images on the fly
Generate lead images
Generation is slow
Generation is hardware intensive

24. EMBEDDINGS

25. WHAT ARE EMBEDDINGS?

27. HOW TO USE THEM IN A CMS
Use an embedding model
Use a multi dimensional vector field
Index the content

28. USE AN EMBEDDING MODEL
$ ollama pull nomic-embed-text

29. CONFIG IN NICK
export const config = {
...
ai: {
models: {
embed: {
name: 'nomic-embed-text',
api: 'http://localhost:11434/api/embed',
dimensions: 768,
minSimilarity: 0.6,
enabled: true,
},
...
},
},
...

30. USE A MULTI DIMENSIONAL
VECTOR FIELD
Install pgvector extension in PostgreSQL
$ CREATE EXTENSION IF NOT EXISTS vector;

31. CREATE INDEX IN NICK
catalog.json
{
"indexes": [
{
"name": "embedding",
"type": "embed",
"attr": "searchableText"
}
],
...
}

32. SAVED VALUE IN DB

33. HOW TO USE THOSE EMBEDDING
VECTORS?

34. SEARCH!
Create an embedding of our search query
Compare the embedding with the index
embeddings
Order by similarity
Limit by similarity

35. SEARCH QUERY IN SQL
SELECT 1 - (_embedding <=> '${embedding}') AS similarity)
FROM catalog
WHERE 1 - (_embedding <=> '${embedding}') > ${minSimilarity}
ORDER BY similarity DESC;

36. SEARCH RESULT

38. IMAGE RECOGNITION

39. In the picture, there's a sleek laptop with its
screen displaying a forest scene. The laptop
appears to be on a table or desk, which also
has some other items like a lamp and a plant
pot in the background. The setting looks
modern and stylish, suggesting that this
image might have been taken for advertising
purposes, showcasing the laptop's design
and features.
WHAT IS IMAGE RECOGNITION?

40. HOW TO USE THEM IN A CMS
Use an image recognition model
Store the generated text
Index the generated text as searchable text
Index the generated text as embedding

41. USE AN IMAGE RECOGNITION
MODEL
$ ollama pull llava

42. CONFIG IN NICK
export const config = {
...
ai: {
models: {
vision: {
name: 'llava',
api: 'http://localhost:11434/api/generate',
enabled: true,
}
...
},
},
...
};

43. IMAGE RECOGNITION

44. RAG
Retrieval Augmented Generation

45. RAG
LLM are trained once, no new information since
LLM's are not aware of "private" data
RAG combines LLM with provided relevant data
Data is provided in the "context"

46. PAGE CONTEXT

47. SITE CONTEXT
Create an embedding of our question
Get relavant documents based on matching
embeddings
Get the text from those documents
Add the text to the context of the query

48. SITE CONTEXT

49. ATTACHMENT CONTEXT

50. HOW CAN THE ASSISTANT HELP
WITH CONTENT EDITING?

51. PAGE EDITING
0:00/ 1:55

52. PROMPTS

53. AI CONTROLPANEL
{
"id": "ai",
"title:i18n": "AI",
"group": "Content",
"schema": {
"fieldsets": [
{
"fields": ["prompts"],
"id": "general",
"title": "General"
}
],
"properties": {
"prompts": {
"title:i18n":"Prompts",

54. ATTACHMENT CONTEXT

55. PROMPTS

56. WHAT ARE TOOLS AND HOW CAN
WE USE THEM?

57. TOOLS
Tools can be used to execute actions by an LLM
Define what they do an what parameters they have
Add them to your chat call
Check the result if a tools is used
Execute the action of the tool

58. EXAMPLE TOOL
'edit page': {
spec: {
type: 'function',
function: {
name: 'edit page',
description: 'Edit the page',
},
},
handler: () => {
document.querySelector('.toolbar-actions .edit').click();
},
message: () => `Editing the page.`,
},

59. EXAMPLE TOOL
'save page': {
spec: {
type: 'function',
function: {
name: 'save page',
description: 'Save the page',
},
},
handler: () => {
document.getElementById('toolbar-save').click();
},
message: () => `The page is saved.`,
},

60. EXAMPLE TOOL
'set title': {
spec: {
type: 'function',
function: {
name: 'set title',
description: 'Sets the title',
parameters: {
type: 'object',
properties: {
title: {
type: 'string',
description: 'The title to be set',
},
},
required:['title'],

61. TOOLS
0:00/ 0:23

62. QUESTIONS?
Want to implement a site using Nick? Talk to me!
slideshare.net/robgietema/how-can-ai-be-used-in-content-management-systems