Using the Splunk Java SDK

damiendallimore 7,128 views 41 slides Sep 13, 2012
Slide 1
Slide 1 of 41
Slide 1
1
Slide 2
2
Slide 3
3
Slide 4
4
Slide 5
5
Slide 6
6
Slide 7
7
Slide 8
8
Slide 9
9
Slide 10
10
Slide 11
11
Slide 12
12
Slide 13
13
Slide 14
14
Slide 15
15
Slide 16
16
Slide 17
17
Slide 18
18
Slide 19
19
Slide 20
20
Slide 21
21
Slide 22
22
Slide 23
23
Slide 24
24
Slide 25
25
Slide 26
26
Slide 27
27
Slide 28
28
Slide 29
29
Slide 30
30
Slide 31
31
Slide 32
32
Slide 33
33
Slide 34
34
Slide 35
35
Slide 36
36
Slide 37
37
Slide 38
38
Slide 39
39
Slide 40
40
Slide 41
41

About This Presentation

Learn the latest and greatest on the Splunk Java SDK. Demos, Code, Code and more Code.


Slide Content

Copyright © 2012 SplunkInc.
Using the SplunkJava SDK
Presented by Damien Dallimore
Developer Evangelist at Splunk

About me
•Developer Evangelist at Splunksince July 2012
•http://dev.splunk.com
•http://splunk-base.splunk.com
•Slides available for my “Splunkingthe JVM” session
•SplunkCommunity Member
•Splunk4JMX
•SplunkJavaLogging
•SplunkBaseAnswers
•SplunkArchitect and Administrator
•Coder, hacker, architect of Enterprise Java solutions around the globe in
many different industries(aviation, core banking, card paymentsetc…)
•Yes, I do have an accent, so please restrain all your sheep, Lord of the
Rings and Kim Dotcom heckles until beer o’clock 
2

Agenda
•Overview of the Splunk Platform
•REST API & SDKs
•Java SDK overview
•Code, Code, Code !
•Thinking outside the Square
•Alternate JVM Languages
•Making it easier for developers to log to Splunk
•Splunkis not just for Production
•Questions (feel free to yell out at any time also)

Accelerate development &
testing with proactive
monitoring
Integrate data from Splunk into
your existing IT environment for
operational visibility
Quickly deliver real-time
business insights from Big Data
outside of IT
4
REST APIs
Splunkd
Custom/Existing
Applications
SDKs
Search, chart and graph
Save and schedule searches as alerts
Export search results
Manage inputs and indexes
Add & remove users and roles
Splunk & Developers
SplunkUI
(Splunk Apps)
Machine Data

REST API & SDKs

What you can do with the SDKs & API
•Integrate with third-party reporting tools and portals
•Log directly to Splunk
•Integrate Splunksearch results into your application
•Extract data for archiving, compliance
•Build a custom UI of your choice
6

SplunkREST API
•Exposes an API method for every feature in the product
•Whatever you can do in the UI –you can do through the API.
•Run searches
•Manage Splunkconfigurations
•API is RESTful
•Endpoints are served by splunkd
•Requests are GET, POST, and DELETE HTTP methods
•Responses are Atom XML Feeds
•JSON coming in 5.0
•Versioning coming in 5.0
•Search results can be output in CSV/JSON/XML
7

Language SDKs
•The SDKs make it easierfor you to use the raw REST API ,
abstracting away much of the lower level plumbing, so you can
instead just focus on developer productivity
•Handling HTTP access
•Authenticating
•Managing namespaces
•Simplifying access to REST endpoints
•Building the correct URL for an endpoint
•Displaying simplified output for searches
•Input of data to a Splunkindex
•Python, Java, Javascriptin beta -Supported
•PHP available now!
•Still study the core REST API though , if you’re anything like me
you like to know what is going on under the hood
8

Java SDK overview

Java SDK Design Principles
•Provide comprehensive coverage of the REST API
•Have a 1:1 mapping of endpoint to class in the SDK
•Provide implementation that felt intuitive to a Java developer
•Lowest common denominator for build –ANT (Any maven people out there?)
•Project support for Eclipse and IntelliJ–to ease getting started
10

Get the Java SDK setup
•Open sourced under the Apache v2.0 license
•Clone from Github: gitclone https://github.com/splunk/splunk-sdk-java.git
•Current release status is “beta”
•Project level support for Eclipse and IntellijIDE’s
•I use Eclipse with the eGitplugin
•Pre-requisites
•JRE 6+
•Ant (builds, javadocgeneration)
•Splunkinstalled
•Run the unit tests and examples
•Setup a “.splunkrc” file in your user’s home directory
•Run an Ant build
•Run examples with the command line wrappers
•Run the Junittests from Ant or within your IDE
11

Key Java SDK Concepts
•Namespaces
•owner: splunkusername
•app: app context
•sharing: user | app | global | system
•Defaults to current user and default app
•Service class
•Instantiate an object to connect and login
•Entry point for REST API calls
•Client/Server state
•Need to maintain state explicitly
•update() : to push changes to splunkd
•refresh() : to get changes from splunkd
12

Java SDK Class Model
13
Service
Resource
ResourceCollection Entity
EntityCollection ApplicationIndex
HTTPService
Input
InputCollection SavedSearchCollection
•Collections use a common mechanism to create and remove entities
•Entities use a common mechanism to retrieve and update property values, and access entity metadata
•Service is a wrapper that facilitates access to all SplunkREST endpoints

public String codeTime(){
return “Lets Rock n Roll”;
}

Connecting / Authenticating
15

Simple Entity Retrieval
16

Logging Events via HTTP REST
17
Uses receivers/simple endpoint Uses receivers/stream endpoint

Logging Events via Raw TCP
18
If you don’t already have a TCP port listening, simply create one via the REST API
Setup Log to Splunk Teardown

Searching Overview
•Search query
•a set of commands and functions you use to retrieve events from an index or a real-time stream , "search *
| head 10".
•Saved search
•a search query that has been saved to be used again and can be set up to run on a regular schedule
•Search job
•an instance of a completed or still-running search operation.Usinga search ID you can access the results of
the search when they become available. Job results are saved for a period of time on the server and can be
retrieved
•Search Modes
•Normal : asynchronous , poll job for status and results
•Blocking : synchronous , a job handle is returned when search is completed
•Oneshot: synchronous , no job handle is returned, results are streamed
•Export : synchronous, not a search per say, doesn’t create a job, results are streamed oldest to newest
Heaps more juicy examples here : http://dev.splunk.com/view/SP-CAAAEHQ
19

Blocking Searches
20
No Job is createdAJob is created

Non-Blocking Search
21

Non-Blocking Search (with Paging)
22
•“maxresultrows” in Splunkconfigdefault 50K
•Not recommended to change this
•If result set > 50K , then page through results

RealtimeSearch
23

Saved Search
24

Processing CSV/JSON/XML results
25

Client/Server State
26

Namespaces
27

Thinking outside the square

Alternate JVM Languages
Scala Groovy Clojure
Javascript(Rhino)JRuby PHP(Quercus)
Ceylon Kotlin Jython
29
We don’t need SDK’s for these languages , we can
just use the Java SDK !

Scala“SDK”
30

Groovy “SDK”
31

SplunkJavaLogging
•A logging framework to allow developers to as seamlessly as
possible integrate Splunkbest practice logging semantics into
their code.
•Custom handler/appenderimplementations(REST and Raw
TCP) for the 3 most prevalent Java logging frameworks in
play. Splunkevents directly from your code.
•LogBack
•Log4j
•java.util.logging
•Implementation of the SPLUNK CIM(Common Information
Model)
32

Developers just log as they are used to
33
2012-08-07 15:54:06:644+1200 name="Failed Login" event_id="someID" app="myapp" user="jane" somefieldname="foobar"
Better
A-HA

Logging Framework takes care of the Splunk
transport , REST or Raw TCP
34

<barf>Typical Java Stacktracesin logs</barf>
35

SplunkJavaLoggingis your friend
36

Java Stacktracesin Splunk
37

Use Splunkin dev/test => better quality
delivered to prod
38
•It’s not good enough to assert that your
software is production ready because the
load test “doesn’t make it fall over”
•Splunkthe app’s machine data throughout
the lifecycle of the test
•Via a SplunkSDK, enrich your test harness
pass/fail assertions with outputs of Splunk
searches
•Catch malignant code tumors that may have
flown under the radar that your harness
alone couldn’t possibly know about.
•Grinder + Java SDK + AssertionsYour
App
Splunk
Java SDK
Testing Tools
Perform test assertions
Execute Tests
SplunkD
REST%/%TCP
Other
Metrics
Universal%
Forwarder
Splunk
Java SDK
REST

Contact Details
Always more than happy to be contacted for questions,
feedback, collaborations, ideas that will change the world etc…
Email : [email protected]
SplunkBase: damiend
Github: damiendallimore
Twitter : @damiendallimore
Blog : http://blogs.splunk.com/dev
SplunkDevPlatform Team : [email protected]
39

Links
Gistsfor all code examples : https://gist.github.com/damiendallimore
Java SDK Homepage : http://dev.splunk.com/view/java-sdk/SP-CAAAECN
Java SDK Githubrepository : https://github.com/splunk/splunk-sdk-java
SplunkJavaLogging: https://github.com/damiendallimore/SplunkJavaLogging
SplunkBest Practice Logging : http://dev.splunk.com/view/logging-best-
practices/SP-CAAADP6
SplunkREST API :
http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTcontents
40

//Thanks for coming !
System.exit(5150);