Christopher Judd: Monitor All Your Stuff with OpenTelemetry and AWS

AWS Community Day Midwest 2024
Christopher M. Judd
Monitor All Your Stuff with OpenTelemetry and AWS

1. Monitor All Your Stuff with
OpenTelemetry and AWS
Christopher M. Juddhttps://bit.ly/adot-awsusmidwest-2024

2. Who thinks architecting, building,
testing, deploying, and monitoring
applications has gotten easier in the
past 10 years?
Cloud
Microservices

3. Christopher M. Judd
CTO and Partner
https://manifestcorp.com/

4. Incontrol theory,observability(o11y) is a measure of how well internal states of
asystemcan be inferred from knowledge of its external outputs.
A system is said to beobservableif, for any possible evolution ofstate and
control vectors, the current state can be estimated using only the information
from outputs (physically, this generally corresponds to information obtained
bysensors)
Observability

5. Observability Pillars
MetricsDistributed TracesLogs

6. Telemetryis thein situcollection of measurementsor other data at remote points
and their automatictransmissionto receiving equipment (telecommunication) for
monitoring.[1]The word is derived from theGreekrootstele, "remote", andmetron,
"measure".
Telemetry

7. javajudd
Software Telemetry Gathers
•Processing Time/Performance
•Application Crashes/Exceptions/Errors
•General Usage Statistics
•Feature Usage
•User Behavior
•Hardware Statistics
•Thread Detection/Investigation

8. Observability
MetricsDistributed TracesLogs

9. Observability
MetricsDistributed TracesLogs

10. Enable X-Ray

11. X-Ray Trace Map

12. X-Ray Traces

13. X-Ray Timeline & Logs

14. CloudWatch - Metrics

16. Blind Spots

17. On Premise
AWS CloudCorporate
data center

18. Containers

19. Instrument application for X-Ray, metrics, & logs
https://docs.aws.amazon.com/xray/latest/devguide/xray-instrumenting-your-app.html

20. https://opentelemetry.io/

21. OpenTelemetry (OTel) provides open source APIs, libraries,
and agents to collect distributed telemetry data (traces,
metrics and logs) for application monitoring

22. Captures
MetricsDistributed TracesLogs

23. Status
MetricsDistributed TracesLogs
Tracing
•API: stable, feature-freeze
•SDK: stable
•Protocol: stable
Metrics
•API: stable
•SDK: mixed
•Protocol: stable
Logging
•API: stable
•SDK: mixed
•Event API: experimental
•Protocol: stable
As of Release v1.34.1

24. Spec Compliance Matrix
https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md

25. OpenTelemetry Support

26. OpenTelemetry Architecture

28. OTLP

29. OTLP

30. OTLP

31. OTLP

32. agent/
collector
otlp
zipkin
prometheus
aws
otlp
otlp
OpenTelemetry Collectors (Agent & Collectors)
OTLP
otlp
otlp
otlp

33. agent/
collector
agent/
collector
agent/
collector
gateway/
collector
otlp
zipkin
prometheus
aws
otlp
otlp
OpenTelemetry Collectors (Agent & Collectors)
OTLP

35. agent/
collector
agent/
collector
agent/
collector
gateway/
collector
otlp
zipkin
prometheus
aws
otlp
otlp
OpenTelemetry Collectors (Agent & Collectors)
OTLP
AWS Distro for OpenTelemetry (ADOT)

36. Demos
https://github.com/spring-petclinic/spring-petclinic-rest
https://github.com/spring-petclinic/spring-petclinic-angular
AngularSpring Boot RESTHQLDB

37. Instrumentation for Java

38. https://github.com/open-telemetry/opentelemetry-java-instrumentation

39. Collectors

40. https://opentelemetry.io/docs/collector/
Collectors

41. AWS Distro for OpenTelemetry
https://aws-otel.github.io/

42. otel-collector-config.yaml
extensions:
health_check:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
cors:
allowed_origins:
-http://*
awsxray:
endpoint: 0.0.0.0:2000
transport: udp
processors:
batch/traces:
timeout: 1s
send_batch_size: 50
batch/metrics:
timeout: 60s
exporters:
awsxray:
awsemf:
awscloudwatchlogs:
log_group_name: "spring-petclinic-rest-logs"
log_stream_name: "spring-petclinic-rest-stream"
service:
pipelines:
traces:
receivers: [otlp,awsxray]
processors: [batch/traces]
exporters: [awsxray]
metrics:
receivers: [otlp]
processors: [batch/metrics]
exporters: [awsemf]
extensions: [health_check]

43. extensions:
health_check:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
cors:
allowed_origins:
-http://*
awsxray:
endpoint: 0.0.0.0:2000
transport: udp
processors:
batch/traces:
timeout: 1s
send_batch_size: 50
batch/metrics:
timeout: 60s
exporters:
awsxray:
awsemf:
awscloudwatchlogs:
log_group_name: "spring-petclinic-rest-logs"
log_stream_name: "spring-petclinic-rest-stream"
service:
pipelines:
traces:
receivers: [otlp,awsxray]
processors: [batch/traces]
exporters: [awsxray]
metrics:
receivers: [otlp]
processors: [batch/metrics]
exporters: [awsemf]
extensions: [health_check]
Receiver 1
Receiver 2
Receiver N
Processor 1Processor 2Processor N
Exporter 1
Exporter 2
Exporter N
Pipelines
pipeline

44. extensions:
health_check:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
cors:
allowed_origins:
-http://*
awsxray:
endpoint: 0.0.0.0:2000
transport: udp
processors:
batch/traces:
timeout: 1s
send_batch_size: 50
batch/metrics:
timeout: 60s
exporters:
awsxray:
awsemf:
awscloudwatchlogs:
log_group_name: "spring-petclinic-rest-logs"
log_stream_name: "spring-petclinic-rest-stream"
service:
pipelines:
traces:
receivers: [otlp,awsxray]
processors: [batch/traces]
exporters: [awsxray]
metrics:
receivers: [otlp]
processors: [batch/metrics]
exporters: [awsemf]
extensions: [health_check]
Receiver 1
Receiver 2
Receiver N
Processor 1Processor 2Processor N
Exporter 1
Exporter 2
Exporter N
pipeline
Receivers

45. extensions:
health_check:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
cors:
allowed_origins:
-http://*
awsxray:
endpoint: 0.0.0.0:2000
transport: udp
processors:
batch/traces:
timeout: 1s
send_batch_size: 50
batch/metrics:
timeout: 60s
exporters:
awsxray:
awsemf:
awscloudwatchlogs:
log_group_name: "spring-petclinic-rest-logs"
log_stream_name: "spring-petclinic-rest-stream"
service:
pipelines:
traces:
receivers: [otlp,awsxray]
processors: [batch/traces]
exporters: [awsxray]
metrics:
receivers: [otlp]
processors: [batch/metrics]
exporters: [awsemf]
extensions: [health_check]
Receiver 1
Receiver 2
Receiver N
Processor 1Processor 2Processor N
Exporter 1
Exporter 2
Exporter N
pipeline
Processors

46. extensions:
health_check:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
cors:
allowed_origins:
-http://*
awsxray:
endpoint: 0.0.0.0:2000
transport: udp
processors:
batch/traces:
timeout: 1s
send_batch_size: 50
batch/metrics:
timeout: 60s
exporters:
awsxray:
awsemf:
awscloudwatchlogs:
log_group_name: "spring-petclinic-rest-logs"
log_stream_name: "spring-petclinic-rest-stream"
service:
pipelines:
traces:
receivers: [otlp,awsxray]
processors: [batch/traces]
exporters: [awsxray]
metrics:
receivers: [otlp]
processors: [batch/metrics]
exporters: [awsemf]
extensions: [health_check]
Receiver 2
Receiver N
Processor 2Processor N
Exporter 1
Exporter 2
Exporter N
pipeline
Exporters
Receiver 1
Processor 1

47. ADOT
otlp
CloudWatch
X-Ray
Docker Compose
docker-compose up -d
https://github.com/cjudd/spring-petclinic-aws-collector
version: "2"
services:
# ADOT Collector
otel-collector:
image: amazon/aws-otel-collector:latest
command: [ "--config=/etc/otel-agent-config.yaml" ]
environment:
-AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
-AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
-AWS_REGION=${AWS_REGION}
volumes:
-./otel-collector-config.yaml:/etc/otel-agent-config.yaml
-~/.aws:/home/aoc/.aws
ports:
-"1777:1777" # pprofextension
-"55679:55679" # zpagesextension
-"4317:4317" # OTLP receiver
-"4318:4318" # HTTP receiver
-"13133" # health_check

48. java -javaagent:opentelemetry-javaagent.jar
-Dotel.traces.exporter=otlp
-Dotel.metrics.exporter=otlp
-Dotel.logs.exporter=otlp
-Dotel.exporter.otlp.endpoint=http://localhost:4318
-Dotel.service.name=spring-petclinic-rest
-jar target/spring-petclinic-rest-x.x.x.jar
ADOT
otlp
CloudWatch
X-Ray
Run Instrumented Java App

49. java -javaagent:opentelemetry-javaagent.jar
-Dotel.service.name=spring-petclinic-rest
-jar target/spring-petclinic-rest-x.x.x.jar
Run Instrumented Java App
ADOT
otlp
CloudWatch
X-Ray

50. Application Traces

51. Application Metrics

52. Resources

53. https://opentelemetry.io/docs/

54. https://github.com/magsther/awesome-opentelemetry

55. email: [email protected]
website: https://juddsolutions.com/
twitter: javajudd
phone: [phone]
email: [email protected]
Christopher M. Judd
linked-in: https://www.linkedin.com/in/christophermjudd/
https://bit.ly/adot-awsusmidwest-2024