一、 Flume安装
1） 准备
jdk:openjdk8
hadoop:hadoop-2.6.0-cdh5.14.0
在cloudera官网下载cdh5版本的flume安装包
$ tar -zxvf flume-ng-1.6.0-cdh5.14.0.tar.gz -C /opt/cloudera/ # 解压到/opt/cloudera目录下
$ cd /opt/cloudera
$ sudo mv flume-ng-1.6.0-cdh5.14.0 flume # 将文件夹名改flume
$ sudo chown -R hadoop ./flume # 修改文件权限
二、 修改配置文件
$ cp conf/flume-env.sh.template conf/flume-env.sh
$ sudo vim conf/flume-env.sh
在配置文件flume-env.sh中，修改JAVA_HOME路径：
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 #换成你的java路径
三、 验证安装是否成功
$ cd /opt/cloudera/flume
$ ./bin/flume-ng version
结果显示如下，则证明安装成功：
Flume 1.6.0-cdh5.14.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 939f6066f6056e7e8042292647eb5032628ac5a0
Compiled by jenkins on Sat Jan 6 13:37:47 PST 2018
From source with checksum c350591306508e3cdd4942fb1debefc9
四、 Flume小例子(Netcat )
任务介绍：NetCat Source：监听一个指定的网络端口，即只要应用程序向这个端口里面写数据，这个source组件就可以获取到信息。
①.创建你自己的属性文件，基于现有的模板。
$ cp conf/flume-conf.properties.template conf/flume.conf
②.修改conf/flume.conf文件
文件内容如下：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 通过channel将source与sink连接起来
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
五、 ubuntu安装telnet
1. 首先更新一下
$ sudo apt-get update
2. 安装
$ sudo apt-get install xinetd telnetd
3.修改/etc/xinetd.conf，并加入以下内容：
# Simple configuration file for xinetd
# Some defaults, and include /etc/xinetd.d/
defaults
{
# Please note that you need a log_type line to be able to use log_on_success
# and log_on_failure. The default is the following :
# log_type = SYSLOG daemon info
instances = 60
log_type = SYSLOG authpriv
log_on_success = HOST PID
log_on_failure = HOST
cps = 25 30
}
includedir /etc/xinetd.d
4.修改/etc/xinetd.d/telnet并加入以下内容：
# default: on
# description: The telnet server serves telnet sessions;it uses
# unencrypted username/password pairs for authentication.
service telnet
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/sbin/in.telnetd
log_on_failure += USERID
}
5.重启机器或重启网络服务
### 1. 重启系统
$ sudo reboot
### 2. 重启网络服务
$ sudo /etc/init.d/xinetd restart
6.测试telnet是否安装成功
### 1. win10下使用cmd窗口执行如下命令：
$ telnet ip地址
六、启动agent的shell操作：
$ cd /opt/cloudera/flume
$ ./bin/flume-ng agent -n a1 -c conf -f conf/flume.conf -Dflume.root.logger=DEBUG,console
结果显示如下,则证明运行成功：
2018-08-13 19:35:41,475 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: []
2018-08-13 19:35:41,476 (conf-file-poller-0) [WARN - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:135)] No configuration found for this host:a1
2018-08-13 19:35:41,492 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
七、使用telnet发送消息
### 查看当前开启的端口号
$ netstat -tnlp
结果显示如下：
tcp6 0 0 127.0.0.1:44444 :::* LISTEN 2124/java
### 监听端口号本机44444
$ telnet localhost 44444
结果显示如下：
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
### 使用telnet进行输入
$ hellow flume
flume 结果显示如下：
2018-08-13 20:35:23,610 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:328)] Chars read = 14
2018-08-13 20:35:23,610 (netcat-handler-0) [DEBUG - org.apache.flume.source.NetcatSource$NetcatSocketHandler.run(NetcatSource.java:332)] Events processed = 1
2018-08-13 20:35:23,610 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 65 6C 6C 6F 77 20 66 6C 75 6D 65 0D hellow flume. }
至此结束。
网友评论