Flume used for logs

In an earlier post, I showed how one may send a stream via netcat to hdfs using flume. Another possibility is to set up a stream that is received by a server whereby the data are directly shown. The idea is that the client starts with telnet or netcat whereby data are sent. On the server side, the data are shown directly.
Let us set up the flume. The config file is:

# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind =
a1.sources.r1.port = 12345
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

Flume is then started with flume-ng agent –conf /etc/flume-ng/conf –conf-file /home/training/netcat_log.conf –name a1 -Dflume.root.logger=INFO,console.

On the client, we start either telnet: telnet 12345. Or we start netcat with: netcat 12345. This looks like:

On the client, we started a stream with “tom”.
Whereas on the serverside, we see:

On the server, this “tom” is received.