Saturday 18 October 2014

Flume Avro Client Example


An AvroSource can be used to receive Avro Events from clients or from other Flume agents in the flow.

flume-ng avro-client sends one or more files to an Avro Source for testing purpose.
Optionally, specify a file containing key/value pairs to set Flume header values.

headers.properties:
timestamp=20141020
hostname=127.0.0.1

$ ./flume-ng avro-client --filename foo.log --headerFile headers.properties \
--host localhost --port 42424
 
Each line of the input will be converted into a single Flume event.

The avro-client is a CLI that sends newline-terminated (non-avro) data to flume. avro-client splits the input file into one event per line.

Consider sending the avro container via some other form of flume source that preserves the full avro container or avro record, e.g. SpoolDirectorySource with BlobDeserializer, HTTPSource with BlobHandler, or a custom flume avro client, etc, per http://flume.apache.org/FlumeUserGuide.html 

Reference:
https://groups.google.com/a/cloudera.org/forum/#!topic/search-user/vngdXym-jjQ

 

No comments:

Post a Comment