JDO 2019: Kubernetes logging techniques with a touch of LogSense - Marcin Stożek

Kubernetes helps us run our applications across multiple nodes using the standardized, declarative way. While we don’t need to think about where our applications are run physically, we still want to have some insights into how they behave. But we are no longer allowed to log into a specific node and just "read the logs" as it does not make much sense. We need to have a proper, automated solution. Kubernetes allows us to use different techniques to achieve this goal. Let's take a look at these techniques and their pros and cons. Once we have those logs, from multiple different applications where each logs things differently – what’s next? This talk also will provide real-life examples of how LogSense works to make sense of all Kubernetes logs, regardless of the format or structure.

1. logging techniqueslogging techniques
with a touch ofwith a touch of
Marcin Stożek "Perk"

2. How apps log thingsHow apps log things
local file
 
remote location
 
stdout/stderr

3. stdout/err loggingstdout/err logging
compliant
 
Problem with multi-line, unstructured logs
Twelve-factor app

4. How Docker manages logsHow Docker manages logs
logging drivers
 
$ docker logs
local
json-file
journald 
docs.docker.com/config/containers/logging/configure/

5. How Docker stores logsHow Docker stores logs
with json-filewith json-file
/var/lib/docker/containers/88f1c.../88f1c...-json.log
{"log":"2019-06-03 12:02:10 +0000 [info]: #0 delayed_timeout is
overwritten by ack_response_timeout\n",
"stream":"stdout", "time":"2019-06-03T12:02:10.252731041Z"}

8. Kubernetes loggingKubernetes logging

9. Logging at the node levelLogging at the node level

10. Logging at the node levelLogging at the node level
✓ logs available through kubectl logs
✓ logs available locally on the node
 
❌this is not really convenient
(but helpful)

13. Cluster-level loggingCluster-level logging
architecturesarchitectures
Exposing logs directly from the application
 
Sidecar container with a logging agent
 
Node-level logging agent on every node
 
Streaming sidecar container

14. Logging from the applicationLogging from the application

15. Logging from the applicationLogging from the application
Push logs directly from the applications
running in the cluster.

16. Logging from the applicationLogging from the application
Push logs directly from the applications
running in the cluster.
Depends on available libraries.
 
Not twelve-factor compliant.
 
Works for multi-line logs.

17. Sidecar containerSidecar container
with a logging agentwith a logging agent

18. Sidecar containerSidecar container
with a logging agentwith a logging agent
Useful when application can log to file only.

19. Sidecar containerSidecar container
with a logging agentwith a logging agent
Same problems as with logging directly from the application.
 
Multi-line logs problem.
 
Who is rotating the log files?
Useful when application can log to file only.

20. Logging with node-agentLogging with node-agent

21. Logging with node-agentLogging with node-agent
Application logs into the stdout.
 
On every node there is a node-agent - DaemonSet.
 
Node-agent takes the logs and pushes them somewhere else.

22. Logging with node-agentLogging with node-agent
Application logs into the stdout.
 
On every node there is a node-agent - DaemonSet.
 
Node-agent takes the logs and pushes them somewhere else.
Logs are still available through kubectl logs command.
 
Problem with multi-line logs.

23. Streaming sidecar containerStreaming sidecar container
github.com/kubernetes/website/pull/14780

24. Streaming sidecar containerStreaming sidecar container
Use when your application logs to file only.
 
Streaming container gets the logs and pushes them to stdout.
 
The rest is the node-agent scenario.

25. Streaming sidecar containerStreaming sidecar container
Use when your application logs to file only.
 
Streaming container gets the logs and pushes them to stdout.
 
The rest is the node-agent scenario.
Multiple sidecars for multiple files.

26. Streaming sidecar containerStreaming sidecar container
Use when your application logs to file only.
 
Streaming container gets the logs and pushes them to stdout.
 
The rest is the node-agent scenario.
Space problem with two log files - inside and outside the pod.
Multiple sidecars for multiple files.

27. spec:
containers:
- name: snowflake
...
volumeMounts:
- name: varlog
mountPath: /var/log
- name: snowflake-log-1
image: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]
volumeMounts:
- name: varlog
mountPath: /var/log
- name: snowflake-log-2
image: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]
volumeMounts:
- name: varlog
mountPath: /var/log
volumes:
- name: varlog
emptyDir: {}1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
github.com/kubernetes/website/blob/master/content/en/examples/admin/logging/two-files-counter-pod-streaming-sidecar.yaml

28. spec:
containers:
- name: snowflake
...
volumeMounts:
- name: varlog
mountPath: /var/log
- name: snowflake-log-1
image: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]
volumeMounts:
- name: varlog
mountPath: /var/log
- name: snowflake-log-2
image: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]
volumeMounts:
- name: varlog
mountPath: /var/log
volumes:
- name: varlog
emptyDir: {}1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
mountPath: /var/logspec:1 containers:2 - name: snowflake3 ...4 volumeMounts:5 - name: varlog6 7 - name: snowflake-log-18 image: busybox9 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]10 volumeMounts:11 - name: varlog12 mountPath: /var/log13 - name: snowflake-log-214 image: busybox15 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]16 volumeMounts:17 - name: varlog18 mountPath: /var/log19 volumes:20 - name: varlog21 emptyDir: {}22
github.com/kubernetes/website/blob/master/content/en/examples/admin/logging/two-files-counter-pod-streaming-sidecar.yaml

29. spec:
containers:
- name: snowflake
...
volumeMounts:
- name: varlog
mountPath: /var/log
- name: snowflake-log-1
image: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]
volumeMounts:
- name: varlog
mountPath: /var/log
- name: snowflake-log-2
image: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]
volumeMounts:
- name: varlog
mountPath: /var/log
volumes:
- name: varlog
emptyDir: {}1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
mountPath: /var/logspec:1 containers:2 - name: snowflake3 ...4 volumeMounts:5 - name: varlog6 7 - name: snowflake-log-18 image: busybox9 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]10 volumeMounts:11 - name: varlog12 mountPath: /var/log13 - name: snowflake-log-214 image: busybox15 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]16 volumeMounts:17 - name: varlog18 mountPath: /var/log19 volumes:20 - name: varlog21 emptyDir: {}22 mountPath: /var/log
mountPath: /var/log
mountPath: /var/logspec:1 containers:2 - name: snowflake3 ...4 volumeMounts:5 - name: varlog6 7 - name: snowflake-log-18 image: busybox9 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]10 volumeMounts:11 - name: varlog12 13 - name: snowflake-log-214 image: busybox15 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]16 volumeMounts:17 - name: varlog18 19 volumes:20 - name: varlog21 emptyDir: {}22
github.com/kubernetes/website/blob/master/content/en/examples/admin/logging/two-files-counter-pod-streaming-sidecar.yaml

30. spec:
containers:
- name: snowflake
...
volumeMounts:
- name: varlog
mountPath: /var/log
- name: snowflake-log-1
image: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]
volumeMounts:
- name: varlog
mountPath: /var/log
- name: snowflake-log-2
image: busybox
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]
volumeMounts:
- name: varlog
mountPath: /var/log
volumes:
- name: varlog
emptyDir: {}1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
mountPath: /var/logspec:1 containers:2 - name: snowflake3 ...4 volumeMounts:5 - name: varlog6 7 - name: snowflake-log-18 image: busybox9 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]10 volumeMounts:11 - name: varlog12 mountPath: /var/log13 - name: snowflake-log-214 image: busybox15 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]16 volumeMounts:17 - name: varlog18 mountPath: /var/log19 volumes:20 - name: varlog21 emptyDir: {}22 mountPath: /var/log
mountPath: /var/log
mountPath: /var/logspec:1 containers:2 - name: snowflake3 ...4 volumeMounts:5 - name: varlog6 7 - name: snowflake-log-18 image: busybox9 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]10 volumeMounts:11 - name: varlog12 13 - name: snowflake-log-214 image: busybox15 args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]16 volumeMounts:17 - name: varlog18 19 volumes:20 - name: varlog21 emptyDir: {}22
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log' ]
args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log' ]spec:1 containers:2 - name: snowflake3 ...4 volumeMounts:5 - name: varlog6 mountPath: /var/log7 - name: snowflake-log-18 image: busybox9 10 volumeMounts:11 - name: varlog12 mountPath: /var/log13 - name: snowflake-log-214 image: busybox15 16 volumeMounts:17 - name: varlog18 mountPath: /var/log19 volumes:20 - name: varlog21 emptyDir: {}22
github.com/kubernetes/website/blob/master/content/en/examples/admin/logging/two-files-counter-pod-streaming-sidecar.yaml

31. $ kubectl logs my-pod snowflake-log-1

0: Mon Jan 1 00:00:00 UTC 2001
1: Mon Jan 1 00:00:01 UTC 2001
2: Mon Jan 1 00:00:02 UTC 2001
...



$ kubectl logs my-pod snowflake-log-2

Mon Jan 1 00:00:00 UTC 2001 INFO 0
Mon Jan 1 00:00:01 UTC 2001 INFO 1
Mon Jan 1 00:00:02 UTC 2001 INFO 2
...1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
github.com/kubernetes/website/blob/master/content/en/examples/admin/logging/two-files-counter-pod-streaming-sidecar.yaml

32. So how to log things?So how to log things?

33. So how to log things?So how to log things?
That depends on the use case, but...

34. So how to log things?So how to log things?
That depends on the use case, but...
? Exposing logs directly from the application
 
? Sidecar container with a logging agent

35. So how to log things?So how to log things?
That depends on the use case, but...
? Exposing logs directly from the application
 
? Sidecar container with a logging agent
✓ Streaming sidecar container

✓✓✓ Node-level logging agent on every node

36. Node-agent logging solutionsNode-agent logging solutions
is your friend here
 
Under like Kubernetes itself
 
 - plug and play
 
Recommended by
FluentD
CNCF
Fluentd Kubernetes Daemonset
k8s fficial documentation

37. kind: DaemonSet
spec:
template:
spec:
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:forward
...
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-forwar d.yaml

38. kind: DaemonSet
spec:
template:
spec:
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:forward
...
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
mountPath: /var/log
mountPath: /var/lib/docker/containers
hostPath:
path: /var/log
hostPath:
path: /var/lib/docker/containerskind: DaemonSet1 spec:2 template:3 spec:4 containers:5 - name: fluentd6 image: fluent/fluentd-kubernetes-daemonset:forward7 ...8 volumeMounts:9 - name: varlog10 11 - name: varlibdockercontainers12 13 readOnly: true14 volumes:15 - name: varlog16 17 18 - name: varlibdockercontainers19 20 21
github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-forwar d.yaml

39. Should we logShould we log
everything?everything?

40. It's not cheapIt's not cheap

41. Logs throttlingLogs throttling
Using Kubernetes means one have many services.
 
We don't use k8s because it's fancy, right?

42. Logs throttlingLogs throttling
Using Kubernetes means one have many services.
 
We don't use k8s because it's fancy, right?

43. Logs throttlingLogs throttling
Using Kubernetes means one have many services.
 
We don't use k8s because it's fancy, right?
Many services means many log messages.

At certain scale it doesn't have much sense to get all the logs.

44. Logs throttlingLogs throttling
Using Kubernetes means one have many services.
 
We don't use k8s because it's fancy, right?
Many services means many log messages.

At certain scale it doesn't have much sense to get all the logs.
FluentD can help with a fluent-plugin-throttle

45. Where to store your logsWhere to store your logs

 
use , eg.
$ helm install stable/elastic-stack
Logging as a ServiceLogSense

46. $ kubectl -n kube-system \
create secret generic logsense-token \
--from-literal=logsense-token=YOUR-LOGSENSE-TOKEN-HERE

$ kubectl apply -f logsense-daemonset.yaml1 2 3 4 5
github.com/logsenseapp/fluentd-kubernetes-daemonset

47. {
"kubernetes":"{
"container_name":"kafka",
"namespace_name":"int",
"pod_name":"kafka-kafka-0",
"container_image":"strimzi/kafka:0.11.1-kafka-2.1.0",
"container_image_id":"docker-pullable://strimzi/kafka@sha256:e741337... ",
"pod_id":"c8eeb49c-67cb-11e9-9b29-12f358b019b2",
"labels":{
"app":"kafka",
"controller-revision-hash ":"kafka-kafka-58dc7cdc78",
"tier":"backend",
"statefulset_kubernetes_io/pod-name ":"kafka-kafka-0",
"strimzi_io/cluster":"kafka",
"strimzi_io/kind":"Kafka",
"strimzi_io/name":"kafka-kafka"
},
"host":"ip-10-100-1-2.ec2.internal",
"master_url":"https://172.20.0.1:443/api",
"namespace_id":"7fe790af-23d2-11e9-8903-0edd77f6b554",
"namespace_labels":{
"name":"int"
}
}",
"docker":"{
"container_id":"e5254e89e508126dbdea587080ec6e01aab660bf62392e41b6e0... "
}",
"stream":"stdout",
"log":" 2019-06-09 18:05:44,774 INFO Deleting segment 271347611 [kafka-scheduler-8] "
}1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

48. {
"kubernetes":"{
"container_name":"kafka",
"namespace_name":"int",
"pod_name":"kafka-kafka-0",
"container_image":"strimzi/kafka:0.11.1-kafka-2.1.0",
"container_image_id":"docker-pullable://strimzi/kafka@sha256:e741337... ",
"pod_id":"c8eeb49c-67cb-11e9-9b29-12f358b019b2",
"labels":{
"app":"kafka",
"controller-revision-hash ":"kafka-kafka-58dc7cdc78",
"tier":"backend",
"statefulset_kubernetes_io/pod-name ":"kafka-kafka-0",
"strimzi_io/cluster":"kafka",
"strimzi_io/kind":"Kafka",
"strimzi_io/name":"kafka-kafka"
},
"host":"ip-10-100-1-2.ec2.internal",
"master_url":"https://172.20.0.1:443/api",
"namespace_id":"7fe790af-23d2-11e9-8903-0edd77f6b554",
"namespace_labels":{
"name":"int"
}
}",
"docker":"{
"container_id":"e5254e89e508126dbdea587080ec6e01aab660bf62392e41b6e0... "
}",
"stream":"stdout",
"log":" 2019-06-09 18:05:44,774 INFO Deleting segment 271347611 [kafka-scheduler-8] "
}1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
"log":" 2019-06-09 18:05:44,774 INFO Deleting segment 271347611 [kafka-scheduler-8] "{1 "kubernetes":"{2 "container_name":"kafka",3 "namespace_name":"int",4 "pod_name":"kafka-kafka-0",5 "container_image":"strimzi/kafka:0.11.1-kafka-2.1.0",6 "container_image_id":"docker-pullable://strimzi/kafka@sha256:e741337... ",7 "pod_id":"c8eeb49c-67cb-11e9-9b29-12f358b019b2",8 "labels":{9 "app":"kafka",10 "controller-revision-hash ":"kafka-kafka-58dc7cdc78",11 "tier":"backend",12 "statefulset_kubernetes_io/pod-name ":"kafka-kafka-0",13 "strimzi_io/cluster":"kafka",14 "strimzi_io/kind":"Kafka",15 "strimzi_io/name":"kafka-kafka"16 },17 "host":"ip-10-100-1-2.ec2.internal",18 "master_url":"https://172.20.0.1:443/api",19 "namespace_id":"7fe790af-23d2-11e9-8903-0edd77f6b554",20 "namespace_labels":{21 "name":"int"22 }23 }",24 "docker":"{25 "container_id":"e5254e89e508126dbdea587080ec6e01aab660bf62392e41b6e0... "26 }",27 "stream":"stdout",28 29 }30

49. {
"kubernetes":"{
"container_name":"kafka",
"namespace_name":"int",
"pod_name":"kafka-kafka-0",
"container_image":"strimzi/kafka:0.11.1-kafka-2.1.0",
"container_image_id":"docker-pullable://strimzi/kafka@sha256:e741337... ",
"pod_id":"c8eeb49c-67cb-11e9-9b29-12f358b019b2",
"labels":{
"app":"kafka",
"controller-revision-hash ":"kafka-kafka-58dc7cdc78",
"tier":"backend",
"statefulset_kubernetes_io/pod-name ":"kafka-kafka-0",
"strimzi_io/cluster":"kafka",
"strimzi_io/kind":"Kafka",
"strimzi_io/name":"kafka-kafka"
},
"host":"ip-10-100-1-2.ec2.internal",
"master_url":"https://172.20.0.1:443/api",
"namespace_id":"7fe790af-23d2-11e9-8903-0edd77f6b554",
"namespace_labels":{
"name":"int"
}
}",
"docker":"{
"container_id":"e5254e89e508126dbdea587080ec6e01aab660bf62392e41b6e0... "
}",
"stream":"stdout",
"log":" 2019-06-09 18:05:44,774 INFO Deleting segment 271347611 [kafka-scheduler-8] "
}1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
"log":" 2019-06-09 18:05:44,774 INFO Deleting segment 271347611 [kafka-scheduler-8] "{1 "kubernetes":"{2 "container_name":"kafka",3 "namespace_name":"int",4 "pod_name":"kafka-kafka-0",5 "container_image":"strimzi/kafka:0.11.1-kafka-2.1.0",6 "container_image_id":"docker-pullable://strimzi/kafka@sha256:e741337... ",7 "pod_id":"c8eeb49c-67cb-11e9-9b29-12f358b019b2",8 "labels":{9 "app":"kafka",10 "controller-revision-hash ":"kafka-kafka-58dc7cdc78",11 "tier":"backend",12 "statefulset_kubernetes_io/pod-name ":"kafka-kafka-0",13 "strimzi_io/cluster":"kafka",14 "strimzi_io/kind":"Kafka",15 "strimzi_io/name":"kafka-kafka"16 },17 "host":"ip-10-100-1-2.ec2.internal",18 "master_url":"https://172.20.0.1:443/api",19 "namespace_id":"7fe790af-23d2-11e9-8903-0edd77f6b554",20 "namespace_labels":{21 "name":"int"22 }23 }",24 "docker":"{25 "container_id":"e5254e89e508126dbdea587080ec6e01aab660bf62392e41b6e0... "26 }",27 "stream":"stdout",28 29 }30
{
"kubernetes":"{
"container_name":"kafka",
"namespace_name":"int",
"pod_name":"kafka-kafka-0",
"container_image":"strimzi/kafka:0.11.1-kafka-2.1.0",
"container_image_id":"docker-pullable://strimzi/kafka@sha256:e741337... ",
"pod_id":"c8eeb49c-67cb-11e9-9b29-12f358b019b2",
"labels":{
"app":"kafka",
"controller-revision-hash ":"kafka-kafka-58dc7cdc78",
"tier":"backend",
"statefulset_kubernetes_io/pod-name ":"kafka-kafka-0",
"strimzi_io/cluster":"kafka",
"strimzi_io/kind":"Kafka",
"strimzi_io/name":"kafka-kafka"
},
"host":"ip-10-100-1-2.ec2.internal",
"master_url":"https://172.20.0.1:443/api",
"namespace_id":"7fe790af-23d2-11e9-8903-0edd77f6b554",
"namespace_labels":{
"name":"int"
}
}",
"docker":"{
"container_id":"e5254e89e508126dbdea587080ec6e01aab660bf62392e41b6e0... "
}",
"stream":"stdout",1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 "log":" 2019-06-09 18:05:44,774 INFO Deleting segment 271347611 [kafka-scheduler-8] "29 }30

52. {
"kubernetes":"{
"container_name":"kafka",
"namespace_name":"int",
"pod_name":"kafka-kafka-0",
"container_image":"strimzi/kafka:0.11.1-kafka-2.1.0",
"container_image_id":"docker-pullable://strimzi/kafka@sha256:e741337... ",
"pod_id":"c8eeb49c-67cb-11e9-9b29-12f358b019b2",
"labels":{
"app":"kafka",
"controller-revision-hash ":"kafka-kafka-58dc7cdc78",
"tier":"backend",
"statefulset_kubernetes_io/pod-name ":"kafka-kafka-0",
"strimzi_io/cluster":"kafka",
"strimzi_io/kind":"Kafka",
"strimzi_io/name":"kafka-kafka"
},
"host":"ip-10-100-1-2.ec2.internal",
"master_url":"https://172.20.0.1:443/api",
"namespace_id":"7fe790af-23d2-11e9-8903-0edd77f6b554",
"namespace_labels":{
"name":"int"
}
}",
"docker":"{
"container_id":"e5254e89e508126dbdea587080ec6e01aab660bf62392e41b6e0... "
}",
"stream":"stdout",
"log":" 2019-06-09 18:05:44,774 INFO Deleting segment 271347611 [kafka-scheduler-8] "
}1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

53. {
"kubernetes":"{
"container_name":"kafka",
"namespace_name":"int",
"pod_name":"kafka-kafka-0",
"container_image":"strimzi/kafka:0.11.1-kafka-2.1.0",
"container_image_id":"docker-pullable://strimzi/kafka@sha256:e741337... ",
"pod_id":"c8eeb49c-67cb-11e9-9b29-12f358b019b2",
"labels":{
"app":"kafka",
"controller-revision-hash ":"kafka-kafka-58dc7cdc78",
"tier":"backend",
"statefulset_kubernetes_io/pod-name ":"kafka-kafka-0",
"strimzi_io/cluster":"kafka",
"strimzi_io/kind":"Kafka",
"strimzi_io/name":"kafka-kafka"
},
"host":"ip-10-100-1-2.ec2.internal",
"master_url":"https://172.20.0.1:443/api",
"namespace_id":"7fe790af-23d2-11e9-8903-0edd77f6b554",
"namespace_labels":{
"name":"int"
}
}",
"docker":"{
"container_id":"e5254e89e508126dbdea587080ec6e01aab660bf62392e41b6e0... "
}",
"stream":"stdout",
"log":" 2019-06-09 18:05:44,774 INFO Deleting segment 271347611 [kafka-scheduler-8] "
}1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
"log":" 2019-06-09 18:05:44,774 INFO Deleting segment 271347611 [kafka-scheduler-8] "{1 "kubernetes":"{2 "container_name":"kafka",3 "namespace_name":"int",4 "pod_name":"kafka-kafka-0",5 "container_image":"strimzi/kafka:0.11.1-kafka-2.1.0",6 "container_image_id":"docker-pullable://strimzi/kafka@sha256:e741337... ",7 "pod_id":"c8eeb49c-67cb-11e9-9b29-12f358b019b2",8 "labels":{9 "app":"kafka",10 "controller-revision-hash ":"kafka-kafka-58dc7cdc78",11 "tier":"backend",12 "statefulset_kubernetes_io/pod-name ":"kafka-kafka-0",13 "strimzi_io/cluster":"kafka",14 "strimzi_io/kind":"Kafka",15 "strimzi_io/name":"kafka-kafka"16 },17 "host":"ip-10-100-1-2.ec2.internal",18 "master_url":"https://172.20.0.1:443/api",19 "namespace_id":"7fe790af-23d2-11e9-8903-0edd77f6b554",20 "namespace_labels":{21 "name":"int"22 }23 }",24 "docker":"{25 "container_id":"e5254e89e508126dbdea587080ec6e01aab660bf62392e41b6e0... "26 }",27 "stream":"stdout",28 29 }30

58. Links, lynx, wget, w3m, curl...Links, lynx, wget, w3m, curl...
kubernetes.io/docs/concepts/cluster-administration/logging/
docs.docker.com/config/containers/logging/
12factor.net/logs
logsense.com

59. logger.info("Thank you!")logger.info("Thank you!")
 /  / @[email protected]