Apache flume how to#
Apache flume software#
Apache flume code#
Apache flume download#

That what I really love about open-source! :DĪctually it was a nice experience, and I got many feedback about the code and style. The funny thing, when I determined to make a pull request to upstream, I found another one opened 2 days before that! But no harm! I just made a PR over that repo to add some futures from code (like filtrating). telegraf -config nf -input-filter flume -testīased on the samples, the output should be the following:įlume_agents_test,type=SOURCE,name=kafka-source01 AppendBatchAcceptedCount=0,AppendReceivedCount=0,KafkaEmpt圜ount=0,OpenConnectionCount=0,StartTime=1516706313621,StopTime=0,AppendAcceptedCount=0,AppendBatchReceivedCount=0,EventAcceptedCount=73353071,EventReceivedCount=73353071,KafkaCommitTimer=6432357,KafkaEventGetTimer=616114808įlume_agents_test,type=CHANNEL,name=hdfs-channel02 EventPutAttemptCount=4234929,EventPutSuccessCount=4234929įlume_agents_test,type=CHANNEL,name=hdfs-channel04 EventPutAttemptCount=12767892,EventPutSuccessCount=12767892įlume_agents_test,type=CHANNEL,name=null-channel EventPutAttemptCount=17057471,EventPutSuccessCount=17057471įlume_agents_test,name=hdfs-sink02,type=SINK BatchUnderflowCount=19281,ConnectionClosedCount=704,ConnectionCreatedCount=705,ConnectionFailedCount=0,EventDrainAttemptCount=4234929,EventDrainSuccessCount=4233252,BatchCompleteCount=691,BatchEmpt圜ount=4573,StartTime=1516706291441,StopTime=0įlume_agents_test,type=SINK,name=hdfs-sink04 BatchEmpt圜ount=36,ConnectionClosedCount=704,ConnectionCreatedCount=707,ConnectionFailedCount=8,StartTime=1516706291445,StopTime=0,BatchCompleteCount=2276,BatchUnderflowCount=2694,EventDrainAttemptCount=12770801,EventDrainSuccessCount=12764502įlume_agents_test,type=CHANNEL,name=hdfs-channel01 EventPutAttemptCount=2757422,EventPutSuccessCount=2757422įlume_agents_test,type=CHANNEL,name=hdfs-channel03 EventPutSuccessCount=36535357,EventPutAttemptCount=36535357įlume_agents_test,type=SINK,name=hdfs-sink01 EventDrainAttemptCount=2757422,EventDrainSuccessCount=2756217,StopTime=0,BatchEmpt圜ount=9466,ConnectionClosedCount=704,ConnectionCreatedCount=705,StartTime=1516706291440,BatchCompleteCount=317,BatchUnderflowCount=22371,ConnectionFailedCount=0įlume_agents_test,type=SINK,name=hdfs-sink03 EventDrainSuccessCount=36533183,StartTime=1516706291444,BatchUnderflowCount=3138,ConnectionClosedCount=704,ConnectionCreatedCount=706,EventDrainAttemptCount=36536896,BatchCompleteCount=7147,BatchEmpt圜ount=124,ConnectionFailedCount=4,StopTime=0įlume_agents_test,type=CHANNEL,name=memChannel EventPutSuccessCount=22948908,EventPutAttemptCount=22948908įlume_agents_test,type=CHANNEL,name=fileChannel EventPutSuccessCount=468085,EventPutAttemptCount=468086 Run the simple http server with samples data: Then compile Telegraf, and generate a test conf:

Apache flume download#

So if you want to try/use this plugin, first download Telegraf if you don't have it, then copy flume directory to the following path:Ĭp -a flume $GOPATH/src//influxdata/telegraf/plugins/inputs Finally I ended up writing one: telegraf-flume-plugin!

Apache flume how to#

I've took a look on some available resources like ' Telegraf contributing guide', and ' How to Write a Telegraf Plugin for Beginners'.

And since I already know enough about Telegraf internals, it wasn't that hard, but to deal with Golang. I found that it's nice to have one, also to polish my Golang knowledge. There are currently two release code lines available, versions 0.9.x and 1.x.

Apache flume software#

Apache Flume is a top-level project at the Apache Software Foundation. So as usual, since we already use for monitoring Telegraf (an agent written in Go for collecting, processing, aggregating, and writing metrics) as part of TICK Stack, so I looked for an official plugin for it, but couldn't find one! Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms.Īfter writing a SaltStack formula for it, then 'monitoring' step came where we monitor every component in our stack. It has a simple and flexible architecture based on streaming data flows.

But first let me quote this from official Flume page:įlume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. _ = oss://$. the beginning of this year while working with Data science team, we had a use case where Apache Flume fits in. Set your_bucket to the bucket for which OSS-HDFS is enabled. The following code provides an example on how to configure a sink: # Configure an OSS sink.