Logstash

how to collect nginxs' accesslogs into elasticsearch by logstash or message-passing.

1. Logstash::Intro
@ARGV

2. Why use Logstash?
•We already have splunk, syslog-ng, chukwa,
graylog2, scribe, flume and so on.
•But we want a free, light-weight and high-
integrality frame for our log:
•non free --> splunk
•heavy java --> scribe,flume
•lose data --> syslog
•non flex --> nxlog

3. How logstash works?
•Ah, just like others, logstash has
input/filter/output plugins.
•Attention: logstash process events, not (only)
loglines!
•"Inputs generate events, filters modify them,
outputs ship them elsewhere." -- [the life of an
event in logstash]
•"events are passed from each phase using
internal queues......Logstash sets each queue
size to 20." -- [the life of an event in logstash]

4. Existing plugins

5. Most popular plugins(inputs)
•amqp
•eventlog
•file
•redis
•stdin
•syslog
•ganglia

6. Most popular plugins(filters)
•date
•grep
•grok
•multiline

7. Most popular plugins(outputs)
•amqp
•elasticsearch
•email
•file
•ganglia
•graphite
•mongodb
•nagios
•redis
•stdout
•zabbix
•websocket

8. Usage in cluster - agent install
•Only an 'all in one' jar download in
http://logstash.net/
•All source include ruby and JRuby in
http://github.com/logstash/
•But we want a lightweight agent in cluster.

9. Usage in cluster - agent install
•Edit Gemfile like:
–source "http://ruby.taobao.org/"
–gem "cabin", "0.4.1"
–gem "bunny"
–gem "uuidtools"
–gem "filewatch", "0.3.3"
•clone logstash/[bin|lib]:
–git clone https://github.com/chenryn/logstash.git
–git branch pure-ruby
•Gem install
–gem install bundler
–bundle
•Run
–ruby logstash/bin/logstash -f logstash/etc/logstash-agent.conf

10. Usage in cluster - agent configuration
–input {
– file {
– type => "nginx"
– path => ["/data/nginx/logs/access.log" ]
– }
–}
–output {
– redis {
– type => "nginx"
– host => "5.5.5.5"
– key => "nginx"
– data_type => "channel"
– }
–}

11. Usage in cluster - server install
•Server is another agent run some filter and
storages.
•Message queue(RabbitMQ is too heavy, Redis
just enough):
–yum install redis-server
–service redis-server start
•Storage: mongo/elasticsearch/Riak
•Visualization: kibana/statsd/riemann/opentsdb
•Run:
–java -jar logstash-1.1.0-monolithic.jar agent -f logstash/etc/server.conf

12. Usage in cluster - server configuration
–input {
– redis {
– type => "nginx"
– host => "5.5.5.5"
– data_type => "channel"
– key => "nginx"
– }
–}
–filter {
– grok {
– type => "nginx"
– pattern => "%{NGINXACCESS}"
– patterns_dir => ["/usr/local/logstash/etc/patterns"]
– }
–}
–output {
– elasticsearch {
– cluster => 'logstash'
– host => '10.5.16.109'
– port => 9300
– }
–}

13. Usage in cluster - grok
•jls-grok is a pattern tool wrote by JRuby
•Lots of examples can be found at:
https://github.com/logstash/logstash/tree/master/patterns
•Here is my "nginx" patterns:
–NGINXURI %{URIPATH}(?:%{URIPARAM})*
–NGINXACCESS \[%{HTTPDATE}\] %{NUMBER:code:int} %{IP:client} %
{HOSTNAME} %{WORD:method} %{NGINXURI:req} %{URIPROTO}/%
{NUMBER:version} %{IP:upstream}(:%{POSINT:port})? %
{NUMBER:upstime:float} %{NUMBER:reqtime:float} %{NUMBER:size:int}
"(%{URIPROTO}://%{HOST:referer}%{NGINXURI:referer}|-)" %
{QS:useragent} "(%{IP:x_forwarder_for}|-)"

14. Usage in cluster - elasticsearch
•ElasticSearch is a production build-on Luence
for the cloud compute.
•more information at:
–http://www.elasticsearch.cn/
•Logstash has an embedded ElasticSearch
already!
•Attention: If you want to build your own
distributed elasticsearch cluster, make sure the
server version is equal to the client used by
logstash!

15. Usage in cluster - elasticsearch
•elasticsearch/config/elasticsearch.yml:
–cluster.name: logstash
–node.name: "ES109"
–node.master: true
–node.data: false
–index.number_of_replicas: 0
–index.number_of_shards: 1
–path.data: /data1/ES/data
–path.logs: /data1/ES/logs
–network.host: 10.5.16.109
–transport.tcp.port: 9300
–transport.tcp.compress: true
–gateway.type: local
–discovery.zen.minimum_master_nodes: 1

16. Usage in cluster - elasticsearch
•The embedded web front for ES is too simple,
sometimes naïve~Try Kibana and EShead.
•https://github.com/rashidkpc/Kibana
•https://github.com/mobz/elasticsearch-head.git
•Attention:there is a bug about ES ---- ifdown
your external network before ES starting and
ifup later.Otherwase your ruby client cannot
connect ES server!

17. Try it please!
•Ah, do not want install,install,install and install?
•Here is a killer application:
–sudo zypper install virtualbox rubygems
–gem install vagrant
–git clone https://github.com/mediatemple/log_wrangler.git
–cd log_wrangler
–PROVISION=1 vagrant up

18. Other output example
•For monitor(example):
–filter {
– grep {
– type => "linux-syslog"
– match => [ "@message","(error|ERROR|CRITICAL)" ]
– add_tag => [ "nagios-update" ]
– add_field => [ "nagios_host", "%{@source_host}", "nagios_service", "the name of your
nagios service check" ]
– }
–}
–output{
– nagios {
– commandfile => “/usr/local/nagios/var/rw/nagios.cmd"
– tags => "nagios-update"
– type => "linux-syslog"
– }
– }

19. Other output example
•For metric
–output {
– statsd {
– increment => "apache.response.%{response}"
– count => [ "apache.bytes", "%{bytes}" ]
– }
–}

20. Advanced Questions
•Is ruby1.8.7 stability enough?
•Try Message::Passing module in CPAN, I love perl~
•Is ElasticSearch high-speedy enough?
•Try Sphinx, see report in ELSA project:
–In designing ELSA, I tried the following components but found them too slow. Here they are ordered from fastest to
slowest for indexing speeds (non-scientifically tested):
1.Tokyo Cabinet
2.MongoDB
3.TokuDB MySQL plugin
4.Elastic Search (Lucene)
5.Splunk
6.HBase
7.CouchDB
8.MySQL Fulltext
•http://code.google.com/p/enterprise-log-search-and-archive/wiki/Documentation#Why_ELSA?

21. Advanced Testing
•How much event/sec can ElasticSearch hold?
•- Logstash::Output::Elasticsearch(HTTP) can only indexes 200+ msg/sec for
one thread.
•- Try _bulk API by myself using perl ElasticSearch::Transport::HTTPLite
module.
•-- speed testing result is 2500+ msg/sec
•-- tesing record see:
http://chenlinux.com/2012/09/16/elasticsearch-bulk-index-speed-testing/
WHY?!

22. Maybe…
•Logstash use an experimental module, we can
see the Logstash::Output::ElasticsearchHTTP
use ftw as http client but it cannot hold bulk size
larger than 200!!
•So we all suggest to use multi-output block in
agent.conf.

23. Advanced ES Settings(1)--problems
•Kibana can search data by using facets APIs.
But when you indexes URLs, they would be
auto-splitted by ‘/’~~
•And search facets at ip from 1000w msgs use
0.1s,but at urls use…ah, timeout!
•When you check your indices size, you will find
that (indices size/indices count) : message
length ~~ 10:1 !!

24. Advanced ES Settings(2)--solution
•Setting ElasticSearch default _mapping
template!
•In fact, ES “store” index data, and then “store”
store data… Yes! If you don’t set “store” : “no”,
all the data reduplicate stored.
•And ES has many analyze plugins.They
automate split words by whitespaces, path
hierachy, keword etc.
•So, set “index”:”not_analyzed” and facets 100k+
URLs can be finished in 1s.

25. Advanced ES Settings(2)--solution
•Optimze:
•Call _optimze API everyday may decrease some
indexed size~
•You can found those solutions in:
•https://github.com/logstash/logstash/wiki/Elasticsearch-Storage-Optimization
•https://github.com/logstash/logstash/wiki/Elasticsearch----Using-index-templates-&-dynamic-mappings

26. Advanced Input -- question
•Now we know how to disable _all field, but there
are still duplicated fields: @fields and
@message!
•Logstash search ES default in @message field
but logstash::Filter::Grok default capture
variables into @fields just from @message!
•How to solve?

27. Advanced Input -- solution
•We know some other systems like
Message::Passing have encode/decode in
addition to input/filter/output.
•In fact logstash has them too~but rename them
as ‘format’.
•So we can define the message format ourself,
just using logformat in nginx.conf.
•(example as follow)

28. Advanced Input -- nginx.conf
–logformat json '{"@timestamp":"$time_iso8601",'
'"@source":"$server_addr",‘
'"@fields":{‘
'"client":"$remote_addr",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,' '"upstreamtime":
$upstream_response_time,'
'"oh":"$upstream_addr",'
'"domain":"$host",'
'"url":"$uri",'
'"status":"$status"}}';
– access_log /data/nginx/logs/access.json json;
•See
http://cookbook.logstash.net/recipes/apache-json-logs/

29. Advanced Input -- json_event
•Now define input block with format:
–input {
– stdin {
– type => "nginx“
– format => "json_event“
– }
–}
•And start in command line:
–tail -F /data/nginx/logs/access.json \
–| sed 's/upstreamtime":-/upstreamtime":0/' \
–| /usr/local/logstash/bin/logstash -f /usr/local/logstash/etc/agent.conf &
•Attention: Upstreamtime may be “-” if status is 400.

30. Advanced Web GUI
•Write your own website using ElasticSearch
RESTful API to search as follows:
–curl -XPOST http://es.domain.com:9200/logstash-2012.09.18/nginx/_search?pretty=1 –d ‘
{
“query”: {
“range”: {
“from”: “now-1h”,
“to”: “now”
}
},
“facets”: {
“curl_test”: {
“date_histogram”: {
“key_field”: “@timestamp”,
“value_field”: “url”,
“interval “: “5m”
}
}
},
“size”: 0
}
’

31. Additional Message::Passing demo
•I do write a demo using Message::Passing,
Regexp::Log, ElasticSearch and so on perl
modules working similar to logstash usage
showed here.
•See:
–http://chenlinux.com/2012/09/16/message-passing-agent/
–http://chenlinux.com/2012/09/16/regexp-log-demo-for-nginx/
–http://chenlinux.com/2012/09/16/message-passing-filter-demo/

32. Reference
•http://logstash.net/docs/1.1.1/tutorials/metrics-from-logs
•http://logwrangler.mtcode.com/
•https://www.virtualbox.org/wiki/Linux_Downloads
•http://vagrantup.com/v1/docs/getting-started/index.html
•http://www.elasticsearch.cn
•http://search.cpan.org/~bobtfish/Message-Passing-
0.010/lib/Message/Passing.pm