DBTA Round Table with Zilliz and Airbyte - Unstructured Data Engineering

DBTA Round Table with Zilliz and Airbyte - Unstructured Data Engineering

https://www.dbta.com/Webinars/2076-Data-Engineering-Best-Practices-for-AI.htm





Data Engineering Best Practices for AI

Data engineering is the backbone of AI systems. After all, the success of AI models heavily depends on the volume, structure, and quality of the data that they rely upon to produce results. With proper tools and practices in place, data engineering can address a number of common challenges that organizations face in deploying and scaling effective AI usage.

Join this October 15th webinar to learn how to:

Quickly integrate data from multiple sources across different environments
Build scalable and efficient data pipelines that can handle large, complex workloads
Ensure that high-quality, relevant data is fed into AI systems
Enhance the performance of AI models with optimized and meaningful input data
Maintain robust data governance, compliance, and security measures
Support real-time AI applications
Reserve your seat today to dive into these issues with our special expert panel.

Register Now to attend the webinar Data Engineering Best Practices for AI. Don't miss this live event on Tuesday, October 15th, 11:00 AM PT / 2:00 PM ET.

1. Introduction to Unstructured Data,
Vector Database and Gen AI
Tim Spann @ Zilliz

2. 2| © Copyright 10/22/23 Zilliz 2| © Copyright 2024 Zilliz
Tim Spann
Principal Developer Advocate
Zilliz
[email protected]
https://www.linkedin.com/in/timothyspann/
https://x.com/PaaSDev

3. 3| © Copyright 9/27/23 Zilliz3| © Copyright 9/27/23 Zilliz
Vector search
is the new
paradigm

4. | © Copyright 9/25/23 Zilliz 4
Retrieval Augmented
Generation RAG
Expand LLMs' knowledge by
incorporating external data sources
into LLMs and your AI applications
Match user behavior or content
features with other similar ones to
make effective recommendations
Recommender System
RecSys
Search for semantically similar
texts across vast amounts of
natural language documents
Text/Semantic Similarity
Search
Molecular Similarity
Search
Search for similar substructures,
superstructures, and other
structures for a specific molecule
Fraud & Anomaly
Detection
Detect data points, events, and
observations that deviate
significantly from the usual pattern
Multimodal Similarity
Search
Search over multiple types of
data simultaneously, e.g. text,
audio, images, video
…powers search across various types of apps

5. | © Copyright 9/25/23 Zilliz 5
Mission:
Helping organizations make sense
of unstructured data.
2017
Founded
$113M
Raised
140
Employees
Redwood City, CA
Headquarters

6. 6 | © Copyright 2024 Zilliz6 6| © Copyright 2024 Zilliz6
Zilliz was built by a top-tier team
of algorithm and database
engineers with a strong
pedigree in developing
high-performance, scalable,
and highly available distributed
systems, uniquely tailored for
vector search.

Built by
database & AI
experts

7. 30K
GitHub
Stars
M
Downloads
250
Contributors
2,600
+Forks
Milvus is an open-source vector database for GenAI projects. docker pull on your
laptop, plug into popular AI dev tools, and push to production with a single line of
code.
Easy Setup

docker pull to
start quickly.
Reusable Code

Write once, and
deploy with one line
of code into the
production
environment
Integration

Plug into OpenAI,
Langchain,
LlamaIndex, and
many more
Feature-rich

Dense & sparse
embeddings,
filtering, reranking
and beyond

8. The Forrester Wave™ Vector
Database Providers, Q3 2024
Zilliz is recognized as the Leader in
the Vector DB Space

9. Data Source: The Digitization of the World by IDC
10%
Other
of newly generated data in 2025
will be unstructured data90%
The world is much more than just text and keywords

10. The Vector Database

11. | © Copyright 2024 Zilliz
Rich functionality
Bulk Import GPU, Intel & ARM
CPU support
Disk Based
Index
Tiered StorageMillion+ level
tenant support
Hybrid Search
Dense & Sparse
RBAC, TLS,
Encryption
Float, Binary, &
Sparse Vector
Tag+Vector
Optimized Filtering
Dynamic Schema

12. Retrieval-Augmented Generation

13. Framework
Hardware
Infrastructure
Embedding Models LLMs
Software Infrastructure
Vector Database
Real-Time AI Stack
Integration

14. 14| © Copyright 10/22/23 Zilliz 14| © Copyright 2024 Zilliz
Milvus
Open Source Self-Managed

Zilliz Cloud
SaaS Fully-Managed

github.com/milvus-io/milvus

Getting Started with Vector Databases
zilliz.com/cloud

15. milvus.io
github.com/milvus-io/
@milvusio
@paasDev


/in/timothyspann
Connect with me! Thank you!