Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rails - Warsaw Ruby Conference 2024

Rails apps can be a black box. Have you ever tried to fix a bug where you just can’t understand what’s going on? This talk will give you practical steps to improve the observability of your Rails app, taking the time to understand and fix defects from hours or days to minutes. Rails 8 will bring an exciting new feature: built-in structured logging. This talk will delve into the transformative impact of structured logging on fixing bugs and saving engineers time. Structured logging, as a cornerstone of observability, offers a powerful way to handle logs compared to traditional text-based logs. This session will guide you through the nuances of structured logging in Rails, demonstrating how it can be used to gain better insights into your application’s behavior. This talk will be a practical, technical deep dive into how to make structured logging work with an existing Rails app.

I talk about the Steps to Observable Software - a practical five step process for improving the observability of your Rails app.

1. Fix Production
Bugs Quickly

The Power Of Structured
Logging In Rails

3. Based on a true story.

4. Based on a true story.
* Names have been changed to
protect the innocent

12. ??????

14. ??????

15. ??????

26. Two Months Later

33. ??????

35. ??????

36. Vicious Cycle

38. How often do you encounter issues where you lack visibility
into your application's performance or errors?

39. How often do you encounter issues where you lack visibility
into your application's performance or errors?

41. ??????

42. ????????????

43. ????????????
??????

44. ????????????
????????????

45. The problem isn't you.

You're not alone.

49. You Want Answered
Question

50. Why are password reset
emails not being sent?
You Want AnsweredQuestion

51. Failing?
Timed Out?
Delayed?
You Want AnsweredQuestion

52. Failing?
Timed Out?
Delayed?
You Want AnsweredQuestion

53. Failing?
Timed Out?
Delayed?
You Want AnsweredQuestion

54. Which jobs are taking time in
within_five_minutes queue?
You Want AnsweredQuestion

55. Data To Collect
Decide

56. Event
Filter By
Group By
Values Of
Data To CollectDecide

57. Event
Filter By
Group By
Values Of
Job Performed



Data To CollectDecide

58. Event
Filter By
Group By
Values Of
Job Performed
Job Queue


Data To CollectDecide

59. Event
Filter By
Group By
Values Of
Job Performed
Job Queue
Job Class

Data To CollectDecide

60. Event
Filter By
Group By
Values Of
Job Performed
Job Queue
Job Class
Duration
Data To CollectDecide

61. Instrumentation
Build

62. InstrumentationBuild
1. Data type (Traces, Logs, Metrics)
2. Plain Text vs Structured
3. Logging Library
4. Customise

63. InstrumentationBuild
Traces
vs
Logs
vs
Metrics

64. InstrumentationBuild
Traces
vs
Logs
vs
Metrics

65. InstrumentationBuild
Plain Text Logs
vs
Structured Logs

66. InstrumentationBuild
Plain Text Logs
vs
Structured Logs

67. Plain Text Logs in Code
InstrumentationBuild

68. Plain Text Logs in UI
InstrumentationBuild

69. Plain Text Logs in UI
InstrumentationBuild

70. Plain Text Logs in UI
InstrumentationBuild
❌

71. Structured Logs in Code
InstrumentationBuild

72. Structured Logs in UI
InstrumentationBuild

73. Structured Logs in UI
InstrumentationBuild

74. Structured Logs in UI
InstrumentationBuild
✅

75. InstrumentationBuild
Logging Library

76. InstrumentationBuild
●Log structured payload
●Integrates with Rails
●Great documentation
●Mature
Criteria

77. InstrumentationBuild
Library Payload Rails Docs Mature
fluentd
lograge
logging
contextual_logger
semantic_logger
dry-logger
Thank you to all authors of these gems for your hard work!

78. InstrumentationBuild
Library Payload Rails Docs Mature
fluentd ❌ ✅ ❌ ✅
lograge ❌ ✅ ❌ ✅
logging ❌ ❌ ❌ ❌
contextual_logger ✅ ✅ ❌ ❌
semantic_logger ✅ ✅ ✅ ✅
dry-logger ✅ ❌ ✅ ❌
Thank you to all authors of these gems for your hard work!

79. InstrumentationBuild
Library Payload Rails Docs Mature
fluentd ❌ ✅ ❌ ✅
lograge ❌ ✅ ❌ ✅
logging ❌ ❌ ❌ ❌
contextual_logger ✅ ✅ ❌ ❌
semantic_logger ✅ ✅ ✅ ✅
dry-logger ✅ ❌ ✅ ❌
Thank you to all authors of these gems for your hard work!

80. InstrumentationBuild
Semantic Logger

81. InstrumentationBuild
Format As JSON

82. InstrumentationBuild
Automatic Logging

83. InstrumentationBuild
●ActionController
●ActionMailer
●ActionView
●ActiveRecord
●ActiveJob
Supported Libraries

84. InstrumentationBuild
ActionController

85. InstrumentationBuild
ActionController

86. InstrumentationBuild
ActionController

87. InstrumentationBuild
ActionController

88. InstrumentationBuild
ActionMailer

89. InstrumentationBuild
ActionMailer

90. InstrumentationBuild
ActionMailer

91. InstrumentationBuild
ActionView

92. InstrumentationBuild
ActiveRecord

93. InstrumentationBuild
ActiveJob

94. InstrumentationBuild
ActiveJob

95. InstrumentationBuild
ActiveJob

96. InstrumentationBuild
Customise

97. 1.No conventions
2.Missing attributes
3.Missing API requests
InstrumentationBuild
Weaknesses

98. InstrumentationBuild
1. No Conventions

99. InstrumentationBuild
job_class
job.class
active_job_class
message.class
job_class_name
job.class.name

100. InstrumentationBuild
Open Telemetry

101. InstrumentationBuild
https://go.joyfulprogramming.com/otel-msg

102. InstrumentationBuild
https://go.joyfulprogramming.com/otel-msg

103. InstrumentationBuild

104. InstrumentationBuild

105. InstrumentationBuild

106. InstrumentationBuild

107. InstrumentationBuild

108. InstrumentationBuild

109. InstrumentationBuild

110. InstrumentationBuild

111. InstrumentationBuild
2. Missing Attributes

112. Request

Headers
Body
ID
User Agent
InstrumentationBuild
Response

Headers
Body

113. Request

Headers
Body
ID
User Agent
InstrumentationBuild
Response

Headers
Body

114. InstrumentationBuild
http://go.joyfulprogramming.com/otel-http

115. InstrumentationBuild
https://go.joyfulprogramming.com/otel-useragent

116. InstrumentationBuild

117. InstrumentationBuild

118. InstrumentationBuild

119. InstrumentationBuild

120. InstrumentationBuild

121. InstrumentationBuild
3. Missing API requests

122. InstrumentationBuild

123. InstrumentationBuild

124. InstrumentationBuild

125. InstrumentationBuild

126. 1.No conventions
2.Missing attributes
3.Missing API requests
InstrumentationBuild
Weaknesses
✅
✅
✅

127. InstrumentationBuild
Send To Logging Tool

128. InstrumentationBuild

131. InstrumentationBuild

134. InstrumentationBuild
Phew!

135. Graphs
Use

136. GraphsUse
Event
Filter By
Group By
Plot Values Of
Job Performed
Job Queue
Job Class
Duration

137. GraphsUse

138. GraphsUse

139. GraphsUse

140. GraphsUse

141. GraphsUse

142. GraphsUse
??????

143. GraphsUse
●Check time range
●Avoid freeform search
●Go wide then narrow down
Tips

144. Improve

145. Improve
Criticise.
Show.
Learn.

146. Improve
Job Latency?
Request ID?
IP?

152. Question

Which jobs are
taking the most time?

160. Answer

Analytics::UpdateUserVisitsJob

161. Question

What's enqueuing
Analytics::UpdateUserVisitsJob ?

169. Answer

profiles#show

170. Question

What IP is hitting
this action?

178. Answer

15.145.1.3

179. Root Cause

Scraper at 15.145.1.3
Hits profiles#show
Floods queue with Analytics job
Delays password reset emails

180. Fix

Block IP in infrastructure

181. Check

Is it fixed?

184. Done!

185. ??????

186. ????????????

187. ????????????
??????

188. ????????????
????????????

189. Results

190. Downtime

191. 98% reduction
Downtime

192. 500 Errors

193. 83% reduction
500 Errors

195. Fix Bugs 20x Faster

196. Extra Resources
https://linktr.ee/joyfulprogramming
Playbook
Sidekiq Course
Newsletter

197. Extra Resources
https://linktr.ee/joyfulprogramming
Workshop - $150
WARSAWRUBY
Expires at 9pm

198. John Gallagher

Thank you!
biggerpockets.com/careers joyfulprogramming.com

199. Questions?
biggerpockets.com/careers joyfulprogramming.com