Telemetry Showdown: Fluent Bit vs. OpenTelemetry Collector by Henrik Rexed

Fluent Bit or OpenTelemetry Collector—which performs better? With OpenTelemetry unifying observability, Fluent Bit has expanded beyond logs. This session explores key differences, benchmark results, and insights to help you choose the best agent for your cloud-native stack.

1. A ScyllaDB Community
Telemetry Showdown:
Fluent Bit vs. OpenTelemetry Collector
A Comprehensive Benchmark Analysis
Henrik Rexed
Cloudnative Advocate

2. Tel????r? S?ow???n
CloudNative Advocate
Henrik Rexed Press start

3. Cloud Native Advocate, CNCF Ambassador
•15+ years of Performance engineering
•Owner of : IsitObservable
Producer of : Perfbytes

Henrik Rexed

4. DISCLAIMER
■No birds or telescopes were harmed in the making of this
presentation

■The intention behind this talk track is not to assign blame to
any CNCF project.

■This session is made to help the community in choosing their
telemetry agent.
4

5. v
Select your fighter
CONFIDENTIAL
5
O?servabiltV
fighte(
Player Select

6. •Understand the difference between fluentbit and the OpenTelemetry
collector by comparing :
•Design Experience
•The plugins for :
•Logs
•Metrics
•Traces

•See various Benchmarking results

•Recommendation on which agents needs to used under specific
conditions



If you stay with me you will ...

7. CONFIDENTIAL 7
Round 1
Design

8. Design Patterns
■Fluentbit ■OpenTelemetry Collector
App Input
Parser
Filter
Observability
Backend
Output
App Receiver
Observability
Backend
Extension
Processor
Exporter
Connector

9. Design Patterns since Fluentbit 2.x
App Input
Parser
Filter
Observability
Backend
Output
App Receiver
Observability
Backend
Extension
Processor
Exporter
Connector
Processor
Filter
Processor
Processor
Filter
■Fluentbit ■OpenTelemetry Collector

10. Design Patterns
App
TRACES METRICS LOGS …
App
TRACES METRICS LOGS
Input
Output
Input
Output
Input
Parser
Filter
Output
Processor
Processor
Processor
Processor
Processor
Processor
■Fluentbit ■OpenTelemetry Collector

11. Fluentbit : Tag is key
■In Fluentbit the pipeline flow rely on Tag

■Input plugins will assign a Tag

■Tag will determine witch content will be routed to
specific filter, parser, output plugins

■Tag is the key concept to define your pipeline flow
11
CONFIDENTIAL 1
1
Input
Filter
Parser
output
Input
Filter
Tag kubelet Tag
kubernetes
Match kubelet Match kubernetes
output
Match kubelet

12. OpenTelemetry Collector: Connector
■The collector provides features helping
us to build complex pipeline. The flow
would be different based on the
content by :

■Building separate pipelines for specific format
■Or taking advantage of the routing connector
( and probably transform)

12
1
2
Receiver
Processor
Routing
output output
Processor
1
Processor
2
Receiver
Processor
output

13. Fluentbit Pipeline format
■Before
13
■After
1
3
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On


[FILTER]
Name Kubernetes
Match kube.*
Merge_Log On
Merge_Log_Trim On


[FILTER]
Name modify
Match kube.*
Rename log content
Rename kubernetes_pod_name k8s.pod.name
Rename kubernetes_namespace_name k8s.namespace.name
Remove time Remove kubernetes_container_hash
Add k8s.cluster.name Onlineboutique


[OUTPUT]
Name http
Match kube.*
host <HOST OF THE DYNATRACE ACTIVE GATE>
port 9999
Retry_Limit false
pipeline:
inputs:
- name: tail
path: /var/log/containers/*.log
multiline.parser: docker, cri
tag: kube.*
mem_Buf_Limit: 5MB
skip_Long_Lines: On
processors:
logs:
- name: modify
match: "kube.*"
add:
- k8s.cluster.name ${CLUSTERNAME}
- dt.kubernetes.cluster.id ${CLUSTER_ID}

filters:
- name: kubernetes
match: kube.*
merge_log: on
keep_log: off
k8s-logging.parser : on
k8S-logging.exclude: on

- name: modify
match: kube.*
rename:
- log content
- kubernetes_pod_name k8s.pod.name
- kubernetes_namespace_name k8s.namespace.name
remove:
- kubernetes_container_image
- kubernetes_container_hash

outputs:
- name: http
host: ${DT_ENDPOINT_HOST}
port: 443
match: “kube.*"
■After

14. Collector Pipeline format
14
CONFIDENTIAL
App Receiver
Processor
Exporter
receivers:
filelog:
include:
- /var/log/pods/*/*/*.log
start_at: beginning
include_file_path: true
include_file_name: false
processors:
batch:
send_batch_max_size: 1000
timeout: 30s
send_batch_size : 800
k8sattributes:
auth_type: "serviceAccount"
passthrough: false
filter:
node_from_env_var: K8S_NODE_NAME
memory_limiter:
check_interval: 1s
limit_percentage: 70
spike_limit_percentage: 30
exporters:
otlphttp:
endpoint: $DT_ENDPOINT/api/v2/otlp
headers:
Authorization: "Api-Token $DT_API_TOKEN"
service:
pipelines:
logs:
receivers: [filelog]
processors: [memory_limiter,k8sattributes ,batch]
exporters: [otlphttp]
telemetry:
metrics:
address: $MY_POD_IP:8888

Extension
Connector

15. Round 2
Logging

16. ■To receive logs we need the following plugins :
■UDP
■TCP
■FluentForward
■OpenTelemetry
■Syslog
■Kafka
■And of course Read from a file






Plugins Required to receive Logs

UDP
TCP
Fluent
Syslog
File
OpenTelemetry
Kafka
Collector 23
log receivers
Fluentbit 17
Log
Receiver

17. ■To process logs we need the following plugins :
■Enrich the logs with:
■Extra resources
■Kubernetes metadata
■Parse the content
■Drop
■Batch the logs
■…etc




Plugins Required to Process Logs

Resource Modify resource
Kubernetes
Filter grep filter
Parse Regexp, json,
parser…etc
Transform (
with OTTL)
Batch Throttle Batch

18. Round 3
Metrics

19. Plugins required to receive metrics
■To receive metrics we mainly need the
following plugins :
■Collectd
■Statsd
■OpenTelemetry
■Prometheus
■Host Metrics ( Windows, Mac, Linux)

Collectd
Statsd
Prometheus
OpenTelemetry
Host

20. Plugins required to Process metrics
■To process metrics we mainly need the following
plugins :
■Enrich the metric with:
■Extra resources
■Kubernetes metadata
■Drop
■Convert

Ressource labels
Transform
Filter metric_selector
CumulativetoDelta
DeltaToRate
kubernetes
Impossible to convert metrics with
Fluentbit

21. Round 4
Traces

22. ■To receive traces we mainly need the following
plugins :
■OpenTelemetry
■Zipkin
■OpenCensus
■Kafka
■..Etc

Plugins required to receive traces

OpenTelemetry
zipkin
OpenCensus
Kafka

23. ■To process traces we mainly need the following
plugins :
■Enrich the trace with:
■Extra resources
■Kubernetes metadata
■Manage the sampling decisions
■Drop
Plugins required to Process traces

Kubernetes
Ressource Content_modifier
TailSampling
probabilisticSampling
Filter
Transform

24. Round 5
Performance

25. The various tests required to compare
■Collector with
processing at
the receiver
■Collecting logs
■Collecting logs , traces
■Collecting Logs, traces
and metrics
■Collector with
processing
after receiving
■Collecting logs
■Collecting logs , traces
■Collecting Logs, traces
and metrics
■Fluentbit
■Collecting logs
■Collecting logs , traces
■Collecting Logs, traces
and metrics

26. Logs
The pipelines
Logs & Traces Logs,Traces and Metrics
Enrich K8s
metadata
Add extra
resources
Export with
OpenTelemetry
Enrich K8s
metadata
Add extra
resources
Export with OpenTelemetry
Enrich K8s
metadata
Add extra
resources
Export with OpenTelemetry
Telemetr
y
metrics
Telemetr
y
metrics

27. Fluentbit Test Architecture
Dynatrace
Kepler
hipstershop Istio
Otel-demo Default
Fluentbit

28. Collector Test Architecture
Dynatrace
Kepler
hipstershop
Istio Otel-demo default

29. Rampup Tests ( 2hours) - Logs
Fluenbit
Collector
Collector
Fluenbit
Fluenbit
Collector
Collector
80
M

30. Rampup Tests ( 2hours) – Logs, Traces
Fluenbit
Collector
Fluenbit
Collector
80
M
Fluenbit
Collector
Collector

31. Rampup Tests ( 2hours) – Logs, Traces & Metrics
Fluenbit
Collector
Fluenbit
Collector
Collector
Fluenbit
Collector
Fluenbit
Collector

32. Soak Test (24h) – Logs, Metrics & Traces

33. ■Collector Soak test with only logs



■Collector soak test with logs and traces:
What signal is causing this memory leek

34. ■By using the pprof extension , we highlight
that the memory leak is related to :
■Metric conversion: Cumulative to delta

■Converting metrics requires to have the right
collector architecture

■Let’s try by using a statefullset collector for
traces and metrics
But what is the actual reason?

35. How can we delegate the metric collection?

36. The Target Allocator
■The OpenTelemtry Operator
provides a Target Allocator

■The Target Allocator take the scrape
config defined in the pipeline and
run it in a separate workload.

■The TargetAllocator create
Prometheus Jobs for each collector
Scrape
config
Target
allocator
Scrape
config
Create
Jobs
Get my
job

37. Collector Test Architecture
Dynatrace
Kepl
er
hipstershop
Istio Otel-demo default
Metric collector

38. Would the target Allocator reduce the memory usage?
■Collector usage



■Target Allocator usage

39. Where should we process our data in the collector?
■When collecting data , it is always recommended to:
■Filter at the source
■Transform local details at source
■Let’s compare the behavior of processing at the receiver vs doing
it with the transform processor:
Filelog Transform

40. Conclusion
40

41. Is it observable
■Looking for educational content on
Observability , Checkout the YouTube Channel :
Is It Observable

42. Thank You