参考:http://shiyanjun.cn/archives/644.html
注:按这篇文章说的做基本上没什么问题,但是有一点这里要说说:
在下载完ext-2.2.zip文件之后,一定要把其解压到libext目录下,然后在执行bin/oozie-setup.sh prepare-war,才能把extJs的东西打入到war包中。否则会报出:Enabling the Oozie Web Console
其他的根据http://shiyanjun.cn/archives/644.html这篇文章操作,经测试没有问题。
![](/assets/blank.gif)
Oozie是一个开源的工作流调度系统,它能够管理逻辑复杂的多个Hadoop作业,按照指定的顺序将其协同运行起来。例如,我们可能有这样一个需求,某个业务系统每天产生20G原始数据,我们每天都要对其进行处理,处理步骤如下所示:
- 通过Hadoop先将原始数据同步到HDFS上;
- 借助MapReduce计算框架对原始数据进行转换,生成的数据以分区表的形式存储到多张Hive表中;
- 需要对Hive中多个表的数据进行JOIN处理,得到一个明细数据Hive大表;
- 将明细数据进行复杂的统计分析,得到排序后的报表信息;
- 需要将统计分析得到的结果数据同步到业务系统中,供业务调用使用。
上述过程可以通过工作流系统来编排任务,最终生成一个工作流实例,然后每天定时启动运行这个实例即可。在这种依赖于Hadoop存储和处理能力要求的应用场景下,Oozie可能能够简化任务调度和执行。
这里,我们在CentOS 6.2系统下安装Oozie-3.3.2,需要安装相关的依赖软件包,下面我们一步一步地进行安装,包括安装配置依赖软件包。这里,我们使用MySQL数据库存储Oozie数据,Hadoop使用的是1.2.1版本。
安装Oozie Server
Oozie Server可以为我们提供很多管理Job的便捷功能,比如,通过可视化界面去管理Job的运行状态,同时也支持我构建含有多个复杂Hadoop Job流程,各个Job之间的依赖关系完全可以通过一个工作流配置文件组装起来,然后由Oozie Server其管理执行。
下载安装,执行如下命令:
1
|
wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.2.1/binaries/apache-maven-3.2.1-bin. tar .gz
|
2
|
tar xvzf apache-maven-3.2.1-bin. tar .gz
|
加入环境变量,使变量配置生效:
1
|
export MAVEN_HOME=/home/shirdrn/cloud/programs/apache-maven-3.2.1
|
2
|
export PATH=$PATH:$MAVEN_HOME/bin
|
安装MySQL数据库,执行如下命令:
1
|
sudo rpm -e --nodeps mysql
|
3
|
sudo yum install -y mysql-server mysql mysql-deve
|
为root用户设置密码:
1
|
mysqladmin -u root password '8YOhyo988_Kjo0'
|
然后可以使用root账号登录MySQL数据库,进行管理:
输入密码登录成功。
下载安装Tomcat Web服务器:
1
|
wget http://apache.dataguru.cn/tomcat/tomcat-7/v7.0.52/bin/apache-tomcat-7.0.52. tar .gz
|
2
|
tar xvzf apache-tomcat-7.0.52. tar .gz
|
设置环境变量:
1
|
export CATALINA_HOME=/home/shirdrn/cloud/programs/apache-tomcat-7.0.52
|
2
|
export PATH=$PATH:$CATALINA_HOME/bin
|
如果使用MySQL存储Oozie数据,需要将MySQL的驱动程序拷贝到Tomcat安装目录下,亦即$CATALINA_HOME/lib下面。
下载ExtJS压缩包:
1
|
wget http://extjs.com/deploy/ext-2.2.zip
|
下载安装,执行如下命令:
1
|
wget http://mirror.bit.edu.cn/apache/oozie/3.3.2/oozie-3.3.2. tar .gz
|
2
|
tar xvzf oozie-3.3.2. tar .gz
|
4
|
bin/mkdistro.sh -DskipTests
|
构建成后,可以在oozie-3.3.2/distro/target目录下看到构建后的文件,例如我的路径是/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2,内容如下所示:
1
|
[shirdrn@oozie-server oozie-3.3.2]$ pwd
|
2
|
/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2
|
3
|
[shirdrn@oozie-server oozie-3.3.2]$ ls
|
4
|
bin lib oozie-core oozie-sharelib-3.3.2. tar .gz
|
5
|
conf libtools oozie-examples. tar .gz oozie.war
|
6
|
docs.zip oozie-client-3.3.2. tar .gz oozie-server release-log.txt
|
将OOZIE_HOME变量指向该目录,修改~/bashrc文件:
1
|
export OOZIE_HOME=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2
|
2
|
export PATH=$PATH:$OOZIE_HOME/bin
|
将ExtJS工具包拷贝到目录$OOZIE_HOME中:
1
|
cp ~/cloud/programs/oozie-3.3.2/ext-2.2.zip $OOZIE_HOME/
|
在上面的目录下创建libext目录,并将hadoop相关的jar库文件拷贝到libext下面,我使用的是Hadoop 1.2.1版本:
1
|
[shirdrn@oozie-server oozie-3.3.2]$ mkdir libext
|
2
|
[shirdrn@oozie-server oozie-3.3.2]$ cp ~/cloud/programs/hadoop-1.2.1/hadoop-*.jar libext/
|
3
|
[shirdrn@oozie-server oozie-3.3.2]$ cp ~/cloud/programs/hadoop-1.2.1/lib/*.jar ./libext/
|
同时,我们使用了MySQL来存储Oozie的元数据,现在需要将MySQL的驱动程序添加到libext目录下:
1
|
cp ~/packages/mysql-connector-java-5.1.29/mysql-connector-java-5.1.29/mysql-connector-java-5.1.29-bin.jar libext/
|
执行下面的命令开始安装:
1
|
bin/oozie-setup.sh prepare-war
|
运行结果,示例如下:
01
|
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
|
03
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/asm-3.2.jar
|
04
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/aspectjrt-1.6.11.jar
|
05
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/aspectjtools-1.6.11.jar
|
06
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-beanutils-1.7.0.jar
|
07
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-beanutils-core-1.8.0.jar
|
08
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-cli-1.2.jar
|
09
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-codec-1.4.jar
|
10
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-collections-3.2.1.jar
|
11
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-configuration-1.6.jar
|
12
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-daemon-1.0.1.jar
|
13
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-digester-1.8.jar
|
14
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-el-1.0.jar
|
15
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-httpclient-3.0.1.jar
|
16
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-io-2.1.jar
|
17
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-lang-2.4.jar
|
18
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-logging-1.1.1.jar
|
19
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-logging-api-1.0.4.jar
|
20
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-math-2.1.jar
|
21
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-net-3.1.jar
|
22
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/core-3.1.1.jar
|
23
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-ant-1.2.1.jar
|
24
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-capacity-scheduler-1.2.1.jar
|
25
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-client-1.2.1.jar
|
26
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-core-1.2.1.jar
|
27
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-examples-1.2.1.jar
|
28
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-fairscheduler-1.2.1.jar
|
29
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-minicluster-1.2.1.jar
|
30
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-test-1.2.1.jar
|
31
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-thriftfs-1.2.1.jar
|
32
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-tools-1.2.1.jar
|
33
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hsqldb-1.8.0.10.jar
|
34
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jackson-core-asl-1.8.8.jar
|
35
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jackson-mapper-asl-1.8.8.jar
|
36
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jasper-compiler-5.5.12.jar
|
37
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jasper-runtime-5.5.12.jar
|
38
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jdeb-0.8.jar
|
39
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jersey-core-1.8.jar
|
40
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jersey-json-1.8.jar
|
41
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jersey-server-1.8.jar
|
42
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jets3t-0.6.1.jar
|
43
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jetty-6.1.26.jar
|
44
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jetty-util-6.1.26.jar
|
45
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jsch-0.1.42.jar
|
46
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/junit-4.5.jar
|
47
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/kfs-0.2.2.jar
|
48
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/log4j-1.2.15.jar
|
49
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/mockito-all-1.8.5.jar
|
50
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/mysql-connector-java-5.1.29-bin.jar
|
51
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/oro-2.0.8.jar
|
52
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/servlet-api-2.5-20081211.jar
|
53
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/slf4j-api-1.4.3.jar
|
54
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/slf4j-log4j12-1.4.3.jar
|
55
|
INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/xmlenc-0.52.jar
|
57
|
New Oozie WAR file with added 'ExtJS library, JARs' at /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie.war
|
60
|
INFO: Oozie is ready to be started
|
这样,上述已经生成了/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie.war文件。
修改conf/oozie-site.xml配置文件,内容如下所示:
02
|
< name >oozie.service.JPAService.jdbc.driver</ name >
|
03
|
< value >com.mysql.jdbc.Driver</ value >
|
09
|
< name >oozie.service.JPAService.jdbc.url</ name >
|
10
|
< value >jdbc:mysql://mysql-server:3306/oozie</ value >
|
16
|
< name >oozie.service.JPAService.jdbc.username</ name >
|
17
|
< value >shirdrn</ value >
|
23
|
< name >oozie.service.JPAService.jdbc.password</ name >
|
27
|
IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,
|
28
|
if empty Configuration assumes it is NULL.
|
默认情况下,Oozie的配置中有个配置项oozie.service.JPAService.create.db.schema,值为false,设置非自动创建数据库,我们保持默认设置,这样可以通过手动创建Oozie数据库,并对其进行权限控制。然后,我们在MySQL数据库中创建数据库,名称为oozie,并进行访问授权:
2
|
GRANT ALL ON oozie.* TO 'shirdrn' @ 'oozie-server' IDENTIFIED BY '0o21e' ;
|
然后可以执行如下命令,生成Oozie所需要的数据表:
1
|
bin/ooziedb.sh create -sqlfile oozie.sql -run
|
查看控制台输出日志,没有报错,并且在当前目录下可以看到,同时也生成了oozie.sql脚本文件。到MySQL数据库中可以看到生成的表,说明上述操作执行成功。
下面可以启动Oozie,使用如下命令:
启动信息,示例如下所示:
01
|
Setting OOZIE_HOME: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2
|
02
|
Setting OOZIE_CONFIG: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf
|
03
|
Sourcing: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-env.sh
|
04
|
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
|
05
|
Setting OOZIE_CONFIG_FILE: oozie-site.xml
|
06
|
Setting OOZIE_DATA: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data
|
07
|
Setting OOZIE_LOG: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs
|
08
|
Setting OOZIE_LOG4J_FILE: oozie-log4j.properties
|
09
|
Setting OOZIE_LOG4J_RELOAD: 10
|
10
|
Setting OOZIE_HTTP_HOSTNAME: oozie-server
|
11
|
Setting OOZIE_HTTP_PORT: 11000
|
12
|
Setting OOZIE_ADMIN_PORT: 11001
|
13
|
Setting OOZIE_HTTPS_PORT: 11443
|
14
|
Setting OOZIE_BASE_URL: http://oozie-server:11000/oozie
|
15
|
Setting CATALINA_BASE: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
|
16
|
Setting OOZIE_HTTPS_KEYSTORE_FILE: /home/shirdrn/.keystore
|
17
|
Setting OOZIE_HTTPS_KEYSTORE_PASS: password
|
18
|
Setting CATALINA_OUT: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/catalina.out
|
19
|
Setting CATALINA_PID: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid
|
21
|
Using CATALINA_OPTS: -Xmx1024m -Dderby.stream.error.file=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/derby.log
|
22
|
Adding to CATALINA_OPTS: -Doozie.home.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2 -Doozie.config.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf -Doozie.log.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs -Doozie.data.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=m1 -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://m1:11000/oozie -Doozie.https.keystore.file=/home/shirdrn/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=
|
24
|
Using CATALINA_BASE: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
|
25
|
Using CATALINA_HOME: /home/shirdrn/cloud/programs/apache-tomcat-7.0.52
|
26
|
Using CATALINA_TMPDIR: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp
|
27
|
Using JRE_HOME: /usr/java/jdk1.7.0_25/
|
28
|
Using CLASSPATH: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/bin/tomcat-juli.jar:/home/shirdrn/cloud/programs/apache-tomcat-7.0.52/bin/bootstrap.jar
|
29
|
Using CATALINA_PID: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid
|
从上面日志可以看到,Oozie管理控制台连接为http://oozie-server:11000/oozie,可以看到图形化界面。
我们的Hadoop平台使用的是用户shirdrn,用户组为shirdrn,这里配置Hadoop代理用户也使用该用户,部署Oozie的主机名为oozie-server。修改Hadoop的配置文件core-site.xml,增加如下配置内容:
3
|
< name >hadoop.proxyuser.shirdrn.hosts</ name >
|
4
|
< value >oozie-server</ value >
|
7
|
< name >hadoop.proxyuser.shirdrn.groups</ name >
|
修改完上述配置后,需要重新启动Hadoop集群才能生效。
安装Oozie Client
我们可以通过在外部的一个Oozie客户端去提交工作流任务,实际上就是一个客户端程序,通过与Oozie Server进行交互,提交任务,并由Oozie Server去调用执行。
我们可以回到前面解压缩Oozie发行包oozie-3.3.2.tar.gz的目录下,通过前面的构建,现在已经可以看到有一个client目录,该目录下就是Oozie的客户端相关文件。含有Oozie客户端脚本的路径,我这里为/home/shirdrn/cloud/programs/oozie-3.3.2/client/target/oozie-client-3.3.2-client/oozie-client-3.3.2。
查看Oozie客户端运行job的命令帮助信息,可以执行如下命令:
1
|
cd /home/shirdrn/cloud/programs/oozie-3.3.2/client/target/oozie-client-3.3.2-client/oozie-client-3.3.2
|
我们可以找到,Oozie发行包中自带的examples,我这里对应的目录是/home/shirdrn/cloud/programs/oozie-3.3.2/examples/target/oozie-examples-3.3.2-examples/examples/apps,我们可以通过运行这些例子来验证安装是否成功。
首先,将Oozie自带的examples上传到HDFS上:
1
|
bin/hadoop fs - mkdir /oozie
|
2
|
bin/hadoop fs -copyFromLocal /home/shirdrn/cloud/programs/oozie-3.3.2/examples/target/oozie-examples-3.3.2-examples/examples /user/shirdrn/examples
|
我们拿examples中的map-reduce来进行验证,修改job.properties文件,配置内容如下所示:
1
|
nameNode=hdfs://m1:9000
|
6
|
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
|
我的环境下,Namenode服务端口为hdfs://m1:9000,JobTracker为m1:19830,运行任务,执行如下命令:
1
|
cd /home/shirdrn/cloud/programs/oozie-3.3.2/client/target/oozie-client-3.3.2-client/oozie-client-3.3.2
|
2
|
bin/oozie job -oozie http://oozie-server:11000/oozie -config /home/shirdrn/cloud/programs/oozie-3.3.2/examples/target/oozie-examples-3.3.2-examples/examples/apps/map-reduce/job.properties -run
|
可以通过OozieWeb管理控制台查看提交运行的任务,如图所示:
![](http://shiyanjun.cn/wp-content/uploads/2014/03/1-oozie-web-console.png)
以及,job配置,运行状态等信息,如图所示:
![](http://shiyanjun.cn/wp-content/uploads/2014/03/2-oozie-job-detail.png)
上面命令选项-run
表示直接运行一个job,当然你可以使用其他选项,如-submit
是提交job,-rerun
是重新运行job,-suspend
是挂起job等等,可以查看命令帮助,或参考相关文档。
参考链接
- http://oozie.apache.org/docs/3.3.2/DG_QuickStart.html
- http://practicalcloudcomputing.com/post/26337621577/installing-and-running-apache-oozie-3-2-x-and-possibly?543b50f0
- http://www.cnblogs.com/cenyuhai/p/3263756.html
Oozie-3.3.2安装配置运行实践相关推荐
- DynaSLAM超详细安装配置运行ubantu20.0.4+opencv2.4.11+tensorflow1.4.0
DynaSLAM超详细安装配置运行ubantu20.0.4+opencv2.4.11+tensorflow1.4.0 注:我现在ubantu的环境是基于可以运行ORB SLAM 2和ORB SLAM3 ...
- Ubuntu20.04安装配置运行DynaSLAM
Ubuntu20.04安装配置运行DynaSLAM 文章目录 Ubuntu20.04安装配置运行DynaSLAM 一.安装Anaconda 二.安装依赖 三.配置Mask_RCNN环境 四.安装Dyn ...
- Pinpoint【环境搭建 01】JDK\HBase\Pinpoint Collector+Web 最新版 2.3.3 安装配置运行验证及脚本文件分享(避坑指南捷径指北)
本文主要是介绍 Pinpoint 环境的部署,小伙伴儿们也可以参考 Pinpoint <官网>的<快速入门>手册,最新版本v2.3.3组件可到官方<GitHub仓库> ...
- Nginx编译-安装-配置-优化实践总结
http://www.zhuxiaodong.net/2016/configure-nginx-server-support-https-http2-on-centos7/ 一些更新说明: 2016- ...
- vscode怎么运行android项目,windows + flutter +android+ vscode 安装配置运行流程(详细版本)...
flutter 是由谷歌发布的一个全新的响应式.跨平台.高性能的移动开发框架,可以快速在iOS和Android上构建高质量的原生用户界面. 框架特点 快速开发:Flutter的热重载可以快速地进行测试 ...
- JDK 安装 配置 运行 帮助文档
JDK 详细安装文档 --安 装 配 置 到 运 行 一.下载JDK 版本-1.8 1.下载jdk 1. 下载地址: http://www.oracle.com/technetwork/java/ja ...
- XMail 安装配置使用
转自:http://blog.sina.com.cn/s/blog_66e177dd0100kwd0.html 前言 企业内部邮件系统可以用于企业内网(局域网)成员,包括人和系统之间通讯.比如一个封闭 ...
- XMail 安装配置使用 (1.27 )
最近在研究xmail,找到了这份资料,在此整理收藏一笔. 原文链接 前言 企业内部邮件系统可以用于企业内网(局域网)成员,包括人和系统之间通讯.比如一个封闭的研发团队,采用代码管理系统(SVN),需要 ...
- 服务器搭建hue_Hue安装配置实践
Hue是一个开源的Apache Hadoop UI系统,最早是由Cloudera Desktop演化而来,由Cloudera贡献给开源社区,它是基于Python Web框架Django实现的.通过使用 ...
最新文章
- SAP收货后更改采购单价格
- Android Hook (2) Java2java
- XPath和lxml类库
- 【计算机网络复习 物理层】2.1.1 物理层基本概念
- firebase登录验证_如何使用Firebase通过三步向身份验证本机添加身份验证
- Pandas数据帧(DataFrame)
- 技术研究院004---数据仓库系统的实现与使用(含OLAP重点讲解)
- css3制作俩面翻转盒子效果
- 技术要求→物理安全→防雷击
- kubernetes failed to start sandbox
- 计算机专业个人职业规划范文200字,计算机专业职业生涯规划范文
- AVATR阿维塔11维修手册电路图技术资料
- java中compar_1.java中Comparor与Comparable的问题
- 计算机快速看图教程,CAD快速看图教程:CAD图纸测量方法集锦
- syswow64删除文件_syswow64,教您syswow64是什么文件夹
- 苹果电脑查看python版本_Mac终端查看python版本号
- 春风十里不如你、与IPC的邂逅
- 联合分析法(Python实现)
- C、C++和C# 到底有什么区别了,纳闷
- Three.js光照贴图添加阴影(·lightMap)
热门文章
- c# 18位身份证源代码
- 基片集成波导原理_ETH研究人员开发微型红外光谱仪 可以集成到芯片当中
- Visual studio 2017基本配置
- Cloudify — 安装部署与基础使用
- NR 5G 协议层服务和功能
- MySQL学习之路:多实例无法启动排错
- 收集国内著名互联网公司前端/UED部门的blog,方便学习交流
- 在C#中使用代理的方式触发事件 (委托和事件 ) (转)
- 如何检测如果滚动条是否可见控件上
- 说说如何搭建 Nginx 反向代理 Tomcat