Open Source TCP or Netflow Log Server Using Graylog

Open Source TCP or Netflow Log Server Using Graylog

1. Open Source TCP or Netflow Log Server Using Graylog

2. Agenda

3. Why we need a log server? Consolidates logs from multiple sources into a single platform. Simplifies log analysis and troubleshooting. Ensures compliance with industry regulations by maintaining audit trails. Enhances performance monitoring and issue diagnosis. Reduces the time and effort required to search and analyze logs. Open Source Solutions: Graylog and ELK Stack, NFsen are excellent open-source options for handling NetFlow data, offering flexibility and powerful visualization. Paid Solutions: like SolarWinds NetFlow Traffic Analyzer, PRTG Network Monitor, and ManageEngine NetFlow Analyzer etc provide advanced features and enterprise support.

4. Sample Network with a log Server

5. Netflow Sample Configuration: Mikrotik

6. TCP logs sample configuration: Mikrotik

7. Overview of Graylog

8. Understanding TCP and Netflow Logs Netflow Logs : NetFlow is a network protocol developed by Cisco for collecting and monitoring IP traffic data. It helps network administrators analyze traffic patterns, monitor performance, and troubleshoot issues. Key Components: NetFlow Exporter: Collects and exports flow data. NetFlow Collector: Receives and stores flow data. NetFlow Analyzer: Analyzes and visualizes the data. NetFlow Versions: NetFlow v5 : Widely used, standard version. NetFlow v9 : Template-based, more flexible and extensible.

9. Understanding TCP and Netflow Logs Here are the most important fields: Source IP Address : The IP address of the device that sent the packets. Destination IP Address : The IP address of the device that received the packets. Source Port : The port number on the source device. Destination Port : The port number on the destination device. Layer 3 Protocol Type : The type of protocol used (e.g., TCP, UDP). Type of Service (ToS) : Quality of service information. Input Interface : The network interface that received the packets. Output Interface : The network interface that sent the packets. Packet Count : The number of packets in the flow. Byte Count : The total number of bytes in the flow. Flow Start Timestamp : The time when the flow started. Flow End Timestamp : The time when the flow ended. Next Hop IP Address : The IP address of the next hop router. Source AS (Autonomous System) : The AS number of the source device. Destination AS : The AS number of the destination device.

10. Understanding TCP and Netflow Logs TCP Logs : TCP logs capture detailed information about TCP connections and their states. These logs help network administrators monitor traffic, diagnose issues, and ensure reliable network communication. Common TCP Flags: SYN (Synchronize): Initiates a connection. ACK (Acknowledgment): Acknowledges received data. FIN (Finish): Indicates the sender is finished sending data. RST (Reset): Abruptly terminates a connection. PSH (Push): Indicates that data should be pushed to the receiving application. URG (Urgent): Indicates that urgent data is being sent.

11. Understanding TCP and Netflow Logs Key Information in TCP Logs: Source IP Address : The IP address of the device that initiated the connection. Destination IP Address : The IP address of the device that received the connection request. Source Port : The port number on the source device. Destination Port : The port number on the destination device. Timestamp : The date and time when the log entry was recorded. TCP Flags : Indicators of the state of the TCP connection, such as SYN, ACK, FIN, and RST. Sequence Numbers : Used to ensure data is transmitted and received in the correct order. Acknowledgment Numbers : Confirm receipt of packets. Window Size : Indicates the amount of data that can be sent before receiving an acknowledgment. Connection State : Describes the current state of the TCP connection (e.g., established, closed).

12. Setting Up Graylog for TCP and Netflow Logs System Requirements: Hardware and software requirements for installing Graylog . Installation Steps: Installing prerequisites (Java, MongoDB, Elasticsearch). Installing Graylog . Configuration: Basic configuration steps for Graylog . Setting up inputs for TCP and Netflow logs. https://go2docs.graylog.org/current/downloading_and_installing_graylog/installing_graylog.html

13. Collecting and Processing Logs Configuring Inputs : Inputs in Graylog are used to receive log messages from various sources. Configuring inputs is the first step in collecting and analyzing log data. 1. Accessing Inputs Navigate to the Graylog web interface. Go to System > Inputs . 2. Selecting Input Type Click on Select input . Choose the type of input based on the log source (e.g., Syslog, NetFlow UDP). 3. Configuring Input Details Fill in the necessary details: Title : Name of the input. Port : The port on which Graylog will listen for incoming messages. Bind Address : The IP address to bind the input to. Additional Parameters : Depending on the input type, additional configurations may be required. 4. Starting the Input Click Launch new input to start the input. Verify that the input is running and receiving messages.

14. Collecting and Processing Logs Example: Configuring a Syslog UDP Input Title: Syslog UDP Port: 514 Bind Address: 0.0.0.0 (all network interfaces) Save and launch the input.

15. Collecting and Processing Logs Log Parsing: Log Parsing is the process of analyzing and extracting meaningful data from log files. It involves converting raw log data into structured information for easier analysis and monitoring. 1. Creating Extractors in Graylog Navigate to System > Inputs . Select the input you want to add an extractor to. Click on Manage Extractors . 2. Defining an Extractor Choose an extractor type (e.g., GROK, JSON, Regex). Define the parsing rule to extract fields from log messages. Test the extractor with sample log data to ensure accuracy. 3. Saving and Using Extractors Save the extractor. Graylog will apply the extractor to incoming log messages on the selected input. Extracted fields will be available for searching and analysis.

16. Collecting and Processing Logs prerouting : in:< pppoe-ram-ruhul@v > out:(unknown 0), connection-state:established,snat proto TCP (ACK,FIN), 10.64.4.28:55842->23.212.164.120:443, NAT (10.64.4.28:55842->103.234.202.0:55842)->23.212.164.120:443 grok_pattern : NAT \(%{ IP:internal_ip }:%{ NUMBER:internal_port }->%{ IP:nat_ip }:%{ NUMBER:nat_port }\)->%{ IP:destination_ip }:%{ NUMBER:destination_port }, len %{ NUMBER:length }

17. Collecting and Processing Logs Log Storage and Retention : 1. Setting Up Log Storage Use Elasticsearch as the primary storage backend for Graylog. Ensure Elasticsearch is configured for high availability and scalability. 2. Configuring Retention Policies in Graylog Navigate to System > Indices . Define retention policies based on index sets. Options include: Time-based retention : Retain logs for a specified period. Size-based retention : Retain logs until a certain index size is reached. Combined retention : Use both time and size-based retention. 3. Implementing Log Rotation Configure index rotation strategies: Time-based rotation : Rotate indices daily, weekly, etc. Size-based rotation : Rotate indices when they reach a specific size. 4. Monitoring Storage and Retention Use Graylog's built-in monitoring tools to track storage usage and retention compliance. Set up alerts for when storage limits are approaching.

18. Collecting and Processing Logs Log Storage and Retention : Example: Rotate indices daily and retain logs for 30 days.

19. Analyzing Logs with Graylog Search and Filter: Search: The ability to query log data to find specific information . Filter: Narrowing down search results to show only relevant log entries. Accessing the Search Interface, Navigate to the Graylog web interface. Click on Search in the top menu. Performing a Basic Search Enter keywords in the search bar. Use common fields like message, source, and timestamp Using Time Range Filters Select a predefined time range (e.g., last 5 minutes, last 24 hours). Customize the time range using the date and time picker.

20. Analyzing Logs with Graylog Example: Filter by log level: level:ERROR Filter by IP address: source_ip:192.168.1.1 Search for login failures: message:login AND message:failed Filter by user or IP address: user:admin

21. Analyzing Logs with Graylog Creating Dashboards : 1. Accessing the Dashboard Interface Navigate to the Graylog web interface. Click on Dashboards in the top menu. 2. Creating a New Dashboard Click on Create new dashboard . Provide a name and description for the dashboard.

22. Analyzing Logs with Graylog

23. Analyzing Logs with Graylog Alerting and Notifications : Automated triggers for specific events or conditions is possible and also notification of those alert via Email/API/Slack is support. Some features are only supported by Graylog Enterprise Edition.

24. Share Dashboard with others Add Users/Teams: Enter the usernames or teams you want to share the dashboard with. Assign Roles: Viewer: Can view the dashboard. Editor: Can edit the dashboard. Owner: Full control over the dashboard, including sharing settings.

25. Use Cases and Examples [TCP flow Dashboard]

26. Use Cases and Examples [TCP flow Dashboard]

27. Use Cases and Examples [ Netflow Dashboard]

28. Use Cases and Examples [Grafana Dashboard]

29. Integration steps for Grafana

30. Best Practices and Tips Security Considerations : Ensuring secure log transmission and storage. Performance Optimization : Tips for optimizing Graylog performance. Community and Support : Leveraging the Graylog community for support and collaboration.

31. Thank You!!!