Setting up Logstash

Posted
Technology:  ,
Services:  ,

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs. This simple example shows how to setup a centralized logstash environment. What we are going to do is install logstash on some servers to gather sample log info. This is the first way in which logstash can be used. These logstash instances “ship” the logs to a centralized logstash server, and are therefor called “shippers”. The centralized server collects all these logs from different shippers. Guess what it is called? Right, the collector.

So far, so simple. And to be honest, it doesn’t get much harder. The configuration of the logstash environment is pretty straightforward. First, java should be installed on all servers which will be sending logs (shippers) and the centralized collector too. Logstash is developed in JRuby and packaged as a jar file. Installing logstash is as simple as downloading the jar and running it.

Installing Java

The servers I used were Debian 6.0 squeeze. Since there is no more official deb package for java in the repositories, I downloaded the JDK7 tarball from oracle and linked the binaries with update-alternatives. Get the right tarball from here (i.e choose the right architecture)

# mkdir /usr/lib/jvm
# tar -xvzf jdk-7u4-linux-x64.gz -C /usr/lib/jvm
# update-alternatives --install /usr/bin/javac javac \
    /usr/lib/jvm/jdk1.7.0_04/bin/javac 1
# update-alternatives --install /usr/bin/java java \
    /usr/lib/jvm/jdk1.7.0_04/bin/java 1
# update-alternatives --install /usr/bin/jar jar \
    /usr/lib/jvm/jdk1.7.0_04/bin/jar 1

To be sure this works you can ask java it’s version :

# java -version

If it is not showing the JDK7, or some other JDK such as open-jdk, you can change this with

# update-alternatives --config javac
# update-alternatives --config java
# update-alternatives --config jar

Java should be installed on all machines on which you want to use logstash.

The collector

Redis

For the collector to fully work, logstash needs some assistance. To receive log messages from different servers, a message broker is used. The broker which the logstash project suggests, is redis . Although redis is more of a datastore, you can use it with logstash to connect the different shippers to.

Get the latest package and extract it.

# wget http://redis.googlecode.com/files/redis-2.4.15.tar.gz
# tar -xzvf redis-2.4.15.tar.gz

Next, we need to make redis. Make sure gcc is installed. If not, you probably want do something like

# apt-get install gcc make

Change to the extracted folder and run make

# cd redis-2.4.15
# make

Now you can run redis. Because we run this in foreground for testing purposes, you probably want to run this in a screen or another terminal.

# ./src/redis-server

You should see following lines :

[21481] 27 Jul 21:01:56 * Server started, Redis version 2.4.15
[21481] 27 Jul 21:01:56 - 0 clients connected (0 slaves), 717496 bytes in use

Downloading the elasticsearch tarball and extracting it

# wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.8.tar.gz
# tar -xzvf elasticsearch-0.19.8.tar.gz

Now you simple can run elasticsearch like this (the -f parameters explicitly says to run in foreground, so agian, use screen or yet another terminal)

# ./elasticsearch-0.19.8/bin/elasticsearch -f

input {
  redis {
    host => "127.0.0.1"
    # these settings should match the output of the agent
    type => "redis-input"
    data_type => "list"
    key => "logstash"
    # We use json_event here since the sender is a logstash agent
    message_format => "json_event"
    }
}
output {
    stdout { debug => true debug_format => "json"}

    elasticsearch { host => "127.0.0.1" }
}

Download the logstash jar and run it

# wget http://semicomplete.com/files/logstash/logstash-1.1.1-monolithic.jar
# java -jar logstash-1.1.1-monolithic.jar agent -f indexer.conf

Note: when logstash gives you an error at this point (which it did with me), you probably need to add the log4j.properties file to the logstash jar file.

According to this issue :

Contents of log4j.properties:

log4j.rootLogger=DEBUG,stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p %d{HH:mm:ss,SSS} %m%n

Add this file to the logstash jar:

# jar -uf logstash-1.0.10.jar log4j.properties

shipper.conf

input {
  stdin {
    type => "stdin-type"
  }
}

output {
  stdout {
    debug => true
    debug_format => "json"
  }
  redis {
    host => "192.168.0.240"
    data_type => "list"
    key => "logstash"
  }
}

Download logstash and run

# wget http://semicomplete.com/files/logstash/logstash-1.1.1-monolithic.jar
# java -jar logstash-1.1.1-monolithic.jar agent -f shipper.conf

# java -jar logstash-1.1.1-monolithic.jar web \
       --backend elasticsearch://127.0.0.1/

Now when you surf to this servers ip address on port 9292 (i.e http://192.168.0.240:9292/) you will land on the logstash webinterface where you can search logs you previously typed in the shipper terminal.

Logstash webui