Adding Telemetry Pipelines to your Platform Toolbox

This talk focuses on Telemetry Pipelines and their potential in controlling log proliferation, on-the-go transforms/processing, and more.

Helpful resources are provided to help readers begin with Telemetry

1. Adding Telemetry
Pipelines to your
Platform toolbox

Bala Ramesh, Sr. TMM
[email protected]

Oct 2024

2. Agenda
01What is a telemetry pipeline
02Why telemetry pipelines make sense
03Demo/Workshop time!
04

05
Resources/Give it a try today

Q/A



22

3. What is a telemetry
pipeline?

4. Telemetry Pipelines give control back to you!
4
❏From the creators of Calyptia and
Fluent Bit
❏Route data from source(s) to
destination(s)
❏Process data (reduce, secure,
enrich)
❏Automate problem detection
❏$$$ savings in log volume reduction

5. Cost increases
5
Easily route log data in hundreds of
formats from any source to any
destination
Process data in-stream to reduce cost,
boost signal, and avoid risk
REDUCE MANAGEMENT OVERHEAD REDUCED DATA STORAGE COSTS INCREASED DEVELOPER PRODUCTIVITY INCREASED PIPELINE RELIABILITY
Solving log data growth and complexity
ANY SOURCE TO ANY DESTINATION TRANSFORM DATA AT THE EDGE
Reduce
●Filter noise
●Prioritize data
●Reduce cost
Enrich
●Add context
●Create governance
Secure
●PII redaction
●Preprocess for SIEM
●Data sovereignty

6. Telemetry Pipeline Architecture
6
Customer Public / Private Cloud
Data Plane
Management Plane
SaaS
Applications
Infrastructure
and Clients
Cloud
Platforms
API Control
Service
User
Console
Meta
Observability
SIEM,
Observability,
Data Lake
Backend
Edge & Agent
Pipelines
Scalable
Pipelines
Kubernetes
Pipelines

7. Why telemetry pipelines
make sense

8. Log data instrumentation, collection, and
routing is complex
Log data growth is out of control
$$$$$
$$$$
$$$
$
(even if the data is useless).
$$
YOUR DATA PRODUCTION
Log data management is complex and expensive
8
2.5x
APPLICATIONS
INFRASTRUCTURE
CLOUD PLATFORMS
DATA GENERATION
VENDOR PROVIDED
INTEGRATION
CUSTOM BUILT
APPLICATION
INGESTING, STORING,
QUERYING

9. Fluent Bit Adoption: 1B → 15B in the last 2 years
Fluent Bit Adoption Fluent Bit Adopters
9

10. Control Cost
20X MORE EFFICIENT
Control Complexity
75% FEWER INCIDENTS
Remediate Faster
Transform logs at the edge to
optimize storage costs and save on
bandwidth
Simplified data ingest and routing.
Improve operational efficiency
from day 2+.
Faster remediation and MTTR by
giving developers the insights they
need in real time
30% MINIMUM COST REDUCTION
The Chronosphere difference
10

11. Demo time!

12. First, the easy one
12
❏https://tinyurl.com/penydemo
❏Install a pipeline on Kubernetes
❏Stream cluster logs
❏Processing rules
❏Forward logs to observability
platform

13. Resources

14. Short link:
https://o11y-workshops.gitlab.io

15. Resources
15
❏https://gitlab.com/o11y-workshops
❏https://chronosphere.io/fluent-bit-academy/
❏1 minute explainer video
❏https://www.youtube.com/@chronosphere-observability/featured
❏https://chronosphere.io/blog-hub/

16. RSVP for the After Party!
The Tippler @ 8 pm

Everyone who
attends will get $25
Uber/Lyft credit

17. Thank You
Bala Ramesh | Sr. TMM
LinkedIn
[email protected]