From Days to Minutes: Telemetry Pipelines for a Stellar Debugging Experience

Telemetry Pipelines route data from multiple sources to multiple destinations, on the fly.

Learn how Telemetry Pipelines can help you control your growing log data! This deck goes over the what, why, and how of telemetry pipelines.

1. From Days to
Minutes: Telemetry
Pipelines for a Stellar
Debugging
Experience
Bala Ramesh, Sr. TMM
[email protected]

Oct 2024

2. Agenda
01What is a telemetry pipeline
02Why telemetry pipelines make sense
03Demo/Workshop time!
04

05
Resources/Give it a try today

Q/A



22

3. What is a telemetry
pipeline?

4. Telemetry Pipelines give control back to you!
4
❏From the creators of Calyptia and
Fluent Bit
❏Route data from source(s) to
destination(s)
❏Process data (reduce, secure,
enrich)
❏Automate problem detection
❏$$$ savings in log volume reduction

5. Cost increases
5
Easily route log data in hundreds of
formats from any source to any
destination
Process data in-stream to reduce cost,
boost signal, and avoid risk
REDUCE MANAGEMENT OVERHEAD REDUCED DATA STORAGE COSTS INCREASED DEVELOPER PRODUCTIVITY INCREASED PIPELINE RELIABILITY
Solving log data growth and complexity
ANY SOURCE TO ANY DESTINATION TRANSFORM DATA AT THE EDGE
Reduce
●Filter noise
●Prioritize data
●Reduce cost
Enrich
●Add context
●Create governance
Secure
●PII redaction
●Preprocess for SIEM
●Data sovereignty

6. Telemetry Pipeline Architecture
6
Customer Public / Private Cloud
Data Plane
Management Plane
SaaS
Applications
Infrastructure
and Clients
Cloud
Platforms
API Control
Service
User
Console
Meta
Observability
SIEM,
Observability,
Data Lake
Backend
Edge & Agent
Pipelines
Scalable
Pipelines
Kubernetes
Pipelines

7. Why telemetry pipelines
make sense

8. Log data instrumentation, collection, and
routing is complex
Log data growth is out of control
$$$$$
$$$$
$$$
$
(even if the data is useless).
$$
YOUR DATA PRODUCTION
Log data management is complex and expensive
8
2.5x
APPLICATIONS
INFRASTRUCTURE
CLOUD PLATFORMS
DATA GENERATION
VENDOR PROVIDED
INTEGRATION
CUSTOM BUILT
APPLICATION
INGESTING, STORING,
QUERYING

9. Fluent Bit Adoption: 1B → 15B in the last 2 years
Fluent Bit Adoption Fluent Bit Adopters
9

10. Control Cost
20X MORE EFFICIENT
Control Complexity
75% FEWER INCIDENTS
Remediate Faster
Transform logs at the edge to
optimize storage costs and save on
bandwidth
Simplified data ingest and routing.
Improve operational efficiency
from day 2+.
Faster remediation and MTTR by
giving developers the insights they
need in real time
30% MINIMUM COST REDUCTION
The Chronosphere difference
10

11. Demo/Workshop
time!

12. First, the easy one
12
❏https://tinyurl.com/devmiodemo
❏Install a pipeline on Kubernetes
❏Stream cluster logs
❏Processing rules
❏Forward logs to observability
platform

13. Next, something more complicated
13
❏https://gitlab.com/o11y-workshops/workshop-fluentbit
❏Multiple labs that walk you through Fluent Bit
❏Self-guided
❏Do it at your own pace!
❏Questions?
❏Email [email protected]

14. Come visit our booth!

15. Resources

16. Resources
16
❏https://gitlab.com/o11y-workshops
❏https://chronosphere.io/fluent-bit-academy/
❏1 minute explainer video
❏https://www.youtube.com/@chronosphere-observability/featured
❏https://chronosphere.io/blog-hub/

17. 17

18. Thank You
Bala Ramesh | Sr. TMM
LinkedIn
[email protected]