Ship Logs by Logstash

Logstash lets you ship, parse and index logs from any source. It works by defining inputs (files, syslog, etc.), filters (grep, split, multiline, etc..) and outputs (elasticsearch, mongodb, etc..). It also provides a UI for accessing and searching your logs.
$./bin/logstash agent -f conf/mylogstash.conf

1. Read files from a folder, then output to stdout in Json format.
Only when the log file is updated, the updates will be fetched and sent by logstash.

input {

  file {
    path  => "/home/cloudera/projects/logstash-1.5.0.beta1/log_sample/*"
    tags => ["aws"]
    start_position => "beginning"
    type => "log"
  }
}

output { stdout { codec => json } }

start_position

Value can be any of: "beginning", "end"
Default value is "end"

Choose where Logstash starts initially reading files: at the beginning or at the end. The default behavior treats files like live streams and thus starts at the end. If you have old data you want to import, set this to ‘beginning’

This option only modifies “first contact” situations where a file is new and not seen before. If a file has already been seen before, this option has no effect.

(1) beginning: When logstash daemon starts it will read the whole file from beginning.

(2) end(default): When logstash deamon ships the data only if the file is changed.

But the issue is:

Every time Logstash fetch the whole contents rather than only the updates. ???

2. Output to Kafka

Since Kafka is integrated into Logstash 1.5 beta, let's use Logstash 1.4.2 with kafka plugin.

Download file from https://github.com/joekiller/logstash-kafka
Then, $ make tarball
Will generate the logstash with Kafka ready to use.

output {
    kafka {
       
        broker_list => "localhost:9092"
        codec => plain {
            format => "%{message}"
        }
        topic_id => "logstash"
        request_required_acks => 1
    }
}


Reference:
http://logstash.net/docs/1.4.2/inputs/file
http://blog.mmlac.com/how-to-pre-process-logs-with-logstash/