Monitoring Far Beyond the Operating System - WeOp 2014

It discusses various aspects such as the implementation of monitoring solutions, the significance of automation in incident management, integration via APIs, prioritization of incidents, and the involvement of senior team members in the implementation and management processes. Additionally, it stresses the importance of revisiting and adjusting processes periodically to ensure effectiveness and adherence. Overall, it underscores the crucial role of both people and processes in maintaining a robust IT environment.

1. Monitoring
Far Beyond the
Operating System
WeOp 2014
Marcus Vechiato - @vechiato
http://weop.com.br

2. Agenda
⦿Goal
⦿How do we envision a monitoring system?
⦿From simple to complex
⦿What to monitor?
⦿What to track?
⦿Locaweb numbers
⦿Where some get lost
⦿Configuration automation
⦿ITIL and ITSM Tools Automatic Incident Creation
⦿Tools already being used
⦿Challenges

3. Goal
The objective of this presentation is to explore monitoring
implementations without focusing on tools.

Best practices highlighting what worked well and the
lessons learned from mistakes made over the years.

4. How do we envision a monitoring system?

5. How do we envision a monitoring system?
⦿It's not just a tool.
⦿The monitoring tool is one of the components of the
process.
⦿Process - it can lead to bureaucracy if it's not effective.

6. Locaweb numbers
⦿Network
⚫Brocade / Cisco / Force10 and others
⦿~21k servers (physical and virtual)
⚫Windows (2003/2008/2012)
⚫Linux (CentOs/Redhat/Debian)
⚫Oracle/MySql/Postgre/MSSQL/MongoDB
⚫VmWare/Xen
⦿~500 thousand items/services monitored every minute
⦿~17 thousand incidents handled per month

7. From simple to complex
⦿Have a clear understanding of your biggest challenges to define your
objectives.
⦿Do not idealize the perfect system that will cover all the gaps, it does not
exist.
⦿Remember: what are your resources and what are the real skills of the
team.
⦿Prefer a gradual implementation with well-defined deliverables.

8. What to monitor?
⦿Core Services and Infrastructure - network/uninterruptible power
supply/temperature/DNS
⦿Operating System (memory/CPU/local network/disk) where applicable
Applications
⚫User perspective (HTTP/TCP requests)
⚫Local (memory usage/threads/processes/etc.)
⦿Business Indicators/errors
⚫Example: Sales per hour Example:
⚫Authentication failures per minute

9. What to track?
Convert the view of infrastructure indicators to products/components/teams

⦿Dashboards for different audiences
⚫Operations
○KPI view by teams/infrastructure
⚫Ex.: MTTR of N1 incidents by priority
⚫Ex.: SLA and MTTR of storage abc
⚫Products/Business
○Common and specific indicator view
⚫Ex.: SLA of product xyz 99.89%
⚫Ex.: MTTR of product xyz 0h45m

10. Where some get lost
⦿It's oversight to diagnose: "the xyz tool doesn't work, we need a new one."
⦿Monitoring probe intervals are too short.
⦿Retries are important to reduce false positives.
⦿From my experience:
⚫Standard probe intervals range from 1 to 5 minutes
⚫Retries:
○5 minutes during deployment/with known instabilities.
○3 minutes in stable environments.

11. Configuration Automation
⦿Monitoring is the best place to start managing component installation and
configurations.
⚫Start with the monitoring agent (if available).
⚫Monitoring server
○Via API where possible
○Configuration files
⦿Which tool to use for automation?
⚫It depends on your environment and the team's knowledge. Chef and
Puppet are good options to start with.

12. ITIL and ITSM Tools
⦿ITSM Tools
⚫I strongly recommend
⚫If you intend to manage incidents automatically, spend more time
evaluating which tool will be used
⦿Processes are the backbone
⚫Incident Management
⚫Problem Management
⚫Change Management
⦿CMDB - registration/control is mandatory
⚫In small installations, your monitoring tool is your CMDB
⚫In larger environments, you will need to synchronize it with the ITSM
tool

13. Automatic Incident Creation
Some benefits of automatic incident creation in larger environments:
⦿Addresses the inefficiency of manual incident logging
⦿Registers failures exactly when they occur
⦿Allows predefining the importance of each component/service and
prioritizing its resolution in case of failure
⦿Reduces informal incident resolution without logging
⦿Provides insight for in-depth analysis of the environment
⦿Integrated with crisis management, reduces resolution time and improves
related communication
⦿Enables realistic calculation of OLAs and SLAs

14. Automatic Incident Creation

⦿Integration via:
⚫API preferably (REST/SOAP)
⚫Email - with templates, most tools allow it (only use as a last resort)
⦿Use the priority when opening the incident to allow prioritization by the
resolving team. According to ITIL, on a scale of 1-5:
⚫Priorities (think of a pyramid):
○1 and 2: should be less than 5% of incidents
○3: 20%
○4: 30%
○5: 45%
⦿For each priority, define different resolution OLAs. Remember that this will
directly affect the size of the team.

15. Automatic Incident Creation

⦿Automatic reopening of incidents if resolved and
continue failing in monitoring or fail again within 30
minutes.
⦿New incident in case of new alarm after 30 minutes
from the last resolved incident.
⦿Suppress incident creation during scheduled
maintenance

16. Automatic Incident Creation

⦿Automatic closure of incidents if monitoring normalizes before team
intervention with status "no intervention" allows:
⚫Refinement of the solution and its efficiency
⚫Adjustment of very tight thresholds
⚫Information for opening Problems
⚫Failures in planning/execution of changes
⚫Quickly resume incident treatment after events with
hundreds/thousands of incidents opened in a short period of time

17. Tools already being used
⦿Monitoring (open source):
⚫Nagios
⚫Check_mk – Locaweb
⚫Zabbix
⦿ITSM:
⚫Service Now (API) – Locaweb
⚫CA – Service Desk Manager (API) – Locaweb
⚫HP – Service Center (API)
⚫OTRS – (API)

18. Challenges

⦿Golden Rule: "Every alarm must have a corrective action" even if it's just
adjusting the thresholds in case of false positives.
⦿Don't be fooled - in the beginning, you will have many false positives.
Persistence is key.
⦿If you don't close incidents automatically during instabilities, typically
network-related, you will be buried in incidents and will miss important
alarms when the instability ceases.

19. Challenges
⦿Who implements the solution and who administers day-to-day operations?
⚫Implementation of the solution: naturally the most Senior team/person.
⚫Who should enable the monitoring in new systems? If you thought in
the intern or the Junior members of the team, you're mistaken. It's also
the responsibility of the most Senior members. It should be automated.

20. Challenges
More important than the tools are the people and adherence to the defined
processes, end-to-end.

Periodically revisit the processes to adjust and evolve according to current
needs.

If any process is not working, change it. Do not allow it to be abandoned or
circumvented.

21. Q&A ?