flume采集练习

采集端口数据到hdfs

环境准备：

1、启动hadoop集群

2、安装flume

3、配置采集方案

agent.sources = s1

agent.channels = c1

agent.sinks = k1

# For each one of the sources, the type is defined

agent.sources.s1.type = netcat

agent.sources.s1.bind = localhost

agent.sources.s1.port = 44444

agent.sources.s1.interceptors= i1

agent.sources.s1.interceptors.i1.type=timestamp

# The channel can be defined as follows.

agent.sources.s1.channels = c1

# Each sink's type must be defined

agent.sinks.k1.type = hdfs

agent.sinks.k1.hdfs.path = hdfs://192.168.1.4:9000/flume/%H%M%S

agent.sinks.k1.hdfs.filePrefix=event-

#Specify the channel the sink should use

agent.sinks.k1.channel = c1

# Each channel's type is defined.

agent.channels.c1.type = memory

# Other config values specific to each type of channel(sink or source)

# can be defined as well

# In this case, it specifies the capacity of the memory channel

agent.channels.c1.capacity = 100

4、启动采集

flume-ng agent -n agent -f flume-conf.properties