参考:http://shiyanjun.cn/archives/644.html

注:按这篇文章说的做基本上没什么问题,但是有一点这里要说说:

在下载完ext-2.2.zip文件之后,一定要把其解压到libext目录下,然后在执行bin/oozie-setup.sh prepare-war,才能把extJs的东西打入到war包中。否则会报出:Enabling the Oozie Web Console

其他的根据http://shiyanjun.cn/archives/644.html这篇文章操作,经测试没有问题。

Oozie是一个开源的工作流调度系统,它能够管理逻辑复杂的多个Hadoop作业,按照指定的顺序将其协同运行起来。例如,我们可能有这样一个需求,某个业务系统每天产生20G原始数据,我们每天都要对其进行处理,处理步骤如下所示:

  1. 通过Hadoop先将原始数据同步到HDFS上;
  2. 借助MapReduce计算框架对原始数据进行转换,生成的数据以分区表的形式存储到多张Hive表中;
  3. 需要对Hive中多个表的数据进行JOIN处理,得到一个明细数据Hive大表;
  4. 将明细数据进行复杂的统计分析,得到排序后的报表信息;
  5. 需要将统计分析得到的结果数据同步到业务系统中,供业务调用使用。

上述过程可以通过工作流系统来编排任务,最终生成一个工作流实例,然后每天定时启动运行这个实例即可。在这种依赖于Hadoop存储和处理能力要求的应用场景下,Oozie可能能够简化任务调度和执行。
这里,我们在CentOS 6.2系统下安装Oozie-3.3.2,需要安装相关的依赖软件包,下面我们一步一步地进行安装,包括安装配置依赖软件包。这里,我们使用MySQL数据库存储Oozie数据,Hadoop使用的是1.2.1版本。

安装Oozie Server

Oozie Server可以为我们提供很多管理Job的便捷功能,比如,通过可视化界面去管理Job的运行状态,同时也支持我构建含有多个复杂Hadoop Job流程,各个Job之间的依赖关系完全可以通过一个工作流配置文件组装起来,然后由Oozie Server其管理执行。

  • 安装Maven构建工具

下载安装,执行如下命令:

1 wget http://mirrors.hust.edu.cn/apache/maven/maven-3/3.2.1/binaries/apache-maven-3.2.1-bin.tar.gz
2 tar xvzf apache-maven-3.2.1-bin.tar.gz

加入环境变量,使变量配置生效:

1 export MAVEN_HOME=/home/shirdrn/cloud/programs/apache-maven-3.2.1
2 export PATH=$PATH:$MAVEN_HOME/bin
  • 安装MySQL数据库

安装MySQL数据库,执行如下命令:

1 sudo rpm -e --nodeps mysql
2 yum list | grep mysql
3 sudo yum install -y mysql-server mysql mysql-deve

为root用户设置密码:

1 mysqladmin -u root password '8YOhyo988_Kjo0'

然后可以使用root账号登录MySQL数据库,进行管理:

1 mysql -u root -p

输入密码登录成功。

  • 安装配置Tomcat

下载安装Tomcat Web服务器:

1 wget http://apache.dataguru.cn/tomcat/tomcat-7/v7.0.52/bin/apache-tomcat-7.0.52.tar.gz
2 tar xvzf apache-tomcat-7.0.52.tar.gz

设置环境变量:

1 export CATALINA_HOME=/home/shirdrn/cloud/programs/apache-tomcat-7.0.52
2 export PATH=$PATH:$CATALINA_HOME/bin

如果使用MySQL存储Oozie数据,需要将MySQL的驱动程序拷贝到Tomcat安装目录下,亦即$CATALINA_HOME/lib下面。

  • 准备ExtJS工具包

下载ExtJS压缩包:

1 wget http://extjs.com/deploy/ext-2.2.zip
  • 安装Oozie

下载安装,执行如下命令:

1 wget http://mirror.bit.edu.cn/apache/oozie/3.3.2/oozie-3.3.2.tar.gz
2 tar xvzf oozie-3.3.2.tar.gz
3 cd oozie-3.3.2
4 bin/mkdistro.sh -DskipTests

构建成后,可以在oozie-3.3.2/distro/target目录下看到构建后的文件,例如我的路径是/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2,内容如下所示:

1 [shirdrn@oozie-server oozie-3.3.2]$ pwd
2 /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2
3 [shirdrn@oozie-server oozie-3.3.2]$ ls
4 bin       lib                        oozie-core             oozie-sharelib-3.3.2.tar.gz
5 conf      libtools                   oozie-examples.tar.gz  oozie.war
6 docs.zip  oozie-client-3.3.2.tar.gz  oozie-server           release-log.txt

将OOZIE_HOME变量指向该目录,修改~/bashrc文件:

1 export OOZIE_HOME=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2
2 export PATH=$PATH:$OOZIE_HOME/bin

将ExtJS工具包拷贝到目录$OOZIE_HOME中:

1 cp ~/cloud/programs/oozie-3.3.2/ext-2.2.zip $OOZIE_HOME/

在上面的目录下创建libext目录,并将hadoop相关的jar库文件拷贝到libext下面,我使用的是Hadoop 1.2.1版本:

1 [shirdrn@oozie-server oozie-3.3.2]$ mkdir libext
2 [shirdrn@oozie-server oozie-3.3.2]$ cp ~/cloud/programs/hadoop-1.2.1/hadoop-*.jar libext/
3 [shirdrn@oozie-server oozie-3.3.2]$ cp ~/cloud/programs/hadoop-1.2.1/lib/*.jar ./libext/

同时,我们使用了MySQL来存储Oozie的元数据,现在需要将MySQL的驱动程序添加到libext目录下:

1 cp ~/packages/mysql-connector-java-5.1.29/mysql-connector-java-5.1.29/mysql-connector-java-5.1.29-bin.jar libext/

执行下面的命令开始安装:

1 bin/oozie-setup.sh prepare-war

运行结果,示例如下:

01   setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
02  
03 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/asm-3.2.jar
04 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/aspectjrt-1.6.11.jar
05 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/aspectjtools-1.6.11.jar
06 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-beanutils-1.7.0.jar
07 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-beanutils-core-1.8.0.jar
08 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-cli-1.2.jar
09 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-codec-1.4.jar
10 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-collections-3.2.1.jar
11 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-configuration-1.6.jar
12 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-daemon-1.0.1.jar
13 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-digester-1.8.jar
14 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-el-1.0.jar
15 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-httpclient-3.0.1.jar
16 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-io-2.1.jar
17 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-lang-2.4.jar
18 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-logging-1.1.1.jar
19 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-logging-api-1.0.4.jar
20 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-math-2.1.jar
21 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/commons-net-3.1.jar
22 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/core-3.1.1.jar
23 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-ant-1.2.1.jar
24 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-capacity-scheduler-1.2.1.jar
25 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-client-1.2.1.jar
26 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-core-1.2.1.jar
27 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-examples-1.2.1.jar
28 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-fairscheduler-1.2.1.jar
29 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-minicluster-1.2.1.jar
30 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-test-1.2.1.jar
31 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-thriftfs-1.2.1.jar
32 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hadoop-tools-1.2.1.jar
33 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/hsqldb-1.8.0.10.jar
34 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jackson-core-asl-1.8.8.jar
35 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jackson-mapper-asl-1.8.8.jar
36 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jasper-compiler-5.5.12.jar
37 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jasper-runtime-5.5.12.jar
38 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jdeb-0.8.jar
39 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jersey-core-1.8.jar
40 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jersey-json-1.8.jar
41 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jersey-server-1.8.jar
42 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jets3t-0.6.1.jar
43 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jetty-6.1.26.jar
44 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jetty-util-6.1.26.jar
45 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/jsch-0.1.42.jar
46 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/junit-4.5.jar
47 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/kfs-0.2.2.jar
48 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/log4j-1.2.15.jar
49 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/mockito-all-1.8.5.jar
50 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/mysql-connector-java-5.1.29-bin.jar
51 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/oro-2.0.8.jar
52 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/servlet-api-2.5-20081211.jar
53 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/slf4j-api-1.4.3.jar
54 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/slf4j-log4j12-1.4.3.jar
55 INFO: Adding extension: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/libext/xmlenc-0.52.jar
56  
57 New Oozie WAR file with added 'ExtJS library, JARs' at /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie.war
58  
59  
60 INFO: Oozie is ready to be started

这样,上述已经生成了/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie.war文件。

  • 配置Oozie

修改conf/oozie-site.xml配置文件,内容如下所示:

01 <property>
02     <name>oozie.service.JPAService.jdbc.driver</name>
03     <value>com.mysql.jdbc.Driver</value>
04     <description>
05         JDBC driver class.
06     </description>
07 </property>
08 <property>
09     <name>oozie.service.JPAService.jdbc.url</name>
10     <value>jdbc:mysql://mysql-server:3306/oozie</value>
11     <description>
12         JDBC URL.
13     </description>
14 </property>
15 <property>
16     <name>oozie.service.JPAService.jdbc.username</name>
17     <value>shirdrn</value>
18     <description>
19         DB user name.
20     </description>
21 </property>
22 <property>
23     <name>oozie.service.JPAService.jdbc.password</name>
24     <value>0o21e</value>
25     <description>
26         DB user password.
27         IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,
28                    if empty Configuration assumes it is NULL.
29     </description>
30 </property>

默认情况下,Oozie的配置中有个配置项oozie.service.JPAService.create.db.schema,值为false,设置非自动创建数据库,我们保持默认设置,这样可以通过手动创建Oozie数据库,并对其进行权限控制。然后,我们在MySQL数据库中创建数据库,名称为oozie,并进行访问授权:

1 CREATE DATABASE oozie;
2 GRANT ALL ON oozie.* TO 'shirdrn'@'oozie-server' IDENTIFIED BY '0o21e';
3 FLUSH PRIVILEGES;

然后可以执行如下命令,生成Oozie所需要的数据表:

1 bin/ooziedb.sh create -sqlfile oozie.sql -run

查看控制台输出日志,没有报错,并且在当前目录下可以看到,同时也生成了oozie.sql脚本文件。到MySQL数据库中可以看到生成的表,说明上述操作执行成功。
下面可以启动Oozie,使用如下命令:

1 bin/oozied.sh start

启动信息,示例如下所示:

01 Setting OOZIE_HOME:          /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2
02 Setting OOZIE_CONFIG:        /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf
03 Sourcing:                    /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-env.sh
04   setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
05 Setting OOZIE_CONFIG_FILE:   oozie-site.xml
06 Setting OOZIE_DATA:          /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data
07 Setting OOZIE_LOG:           /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs
08 Setting OOZIE_LOG4J_FILE:    oozie-log4j.properties
09 Setting OOZIE_LOG4J_RELOAD:  10
10 Setting OOZIE_HTTP_HOSTNAME: oozie-server
11 Setting OOZIE_HTTP_PORT:     11000
12 Setting OOZIE_ADMIN_PORT:     11001
13 Setting OOZIE_HTTPS_PORT:     11443
14 Setting OOZIE_BASE_URL:      http://oozie-server:11000/oozie
15 Setting CATALINA_BASE:       /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
16 Setting OOZIE_HTTPS_KEYSTORE_FILE:     /home/shirdrn/.keystore
17 Setting OOZIE_HTTPS_KEYSTORE_PASS:     password
18 Setting CATALINA_OUT:        /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/catalina.out
19 Setting CATALINA_PID:        /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid
20  
21 Using   CATALINA_OPTS:        -Xmx1024m -Dderby.stream.error.file=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/derby.log
22 Adding to CATALINA_OPTS:     -Doozie.home.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2 -Doozie.config.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf -Doozie.log.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs -Doozie.data.dir=/home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=m1 -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://m1:11000/oozie -Doozie.https.keystore.file=/home/shirdrn/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=
23  
24 Using CATALINA_BASE:   /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
25 Using CATALINA_HOME:   /home/shirdrn/cloud/programs/apache-tomcat-7.0.52
26 Using CATALINA_TMPDIR: /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp
27 Using JRE_HOME:        /usr/java/jdk1.7.0_25/
28 Using CLASSPATH:       /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/bin/tomcat-juli.jar:/home/shirdrn/cloud/programs/apache-tomcat-7.0.52/bin/bootstrap.jar
29 Using CATALINA_PID:    /home/shirdrn/cloud/programs/oozie-3.3.2/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid

从上面日志可以看到,Oozie管理控制台连接为http://oozie-server:11000/oozie,可以看到图形化界面。

  • 整合Oozie和Hadoop

我们的Hadoop平台使用的是用户shirdrn,用户组为shirdrn,这里配置Hadoop代理用户也使用该用户,部署Oozie的主机名为oozie-server。修改Hadoop的配置文件core-site.xml,增加如下配置内容:

1 <!-- OOZIE -->
2 <property>
3      <name>hadoop.proxyuser.shirdrn.hosts</name>
4      <value>oozie-server</value>
5 </property>
6 <property>
7      <name>hadoop.proxyuser.shirdrn.groups</name>
8      <value>shirdrn</value>
9 </property>

修改完上述配置后,需要重新启动Hadoop集群才能生效。

安装Oozie Client

我们可以通过在外部的一个Oozie客户端去提交工作流任务,实际上就是一个客户端程序,通过与Oozie Server进行交互,提交任务,并由Oozie Server去调用执行。
我们可以回到前面解压缩Oozie发行包oozie-3.3.2.tar.gz的目录下,通过前面的构建,现在已经可以看到有一个client目录,该目录下就是Oozie的客户端相关文件。含有Oozie客户端脚本的路径,我这里为/home/shirdrn/cloud/programs/oozie-3.3.2/client/target/oozie-client-3.3.2-client/oozie-client-3.3.2。
查看Oozie客户端运行job的命令帮助信息,可以执行如下命令:

1 cd /home/shirdrn/cloud/programs/oozie-3.3.2/client/target/oozie-client-3.3.2-client/oozie-client-3.3.2
2 bin/oozie help
3 bin/oozie help job

我们可以找到,Oozie发行包中自带的examples,我这里对应的目录是/home/shirdrn/cloud/programs/oozie-3.3.2/examples/target/oozie-examples-3.3.2-examples/examples/apps,我们可以通过运行这些例子来验证安装是否成功。
首先,将Oozie自带的examples上传到HDFS上:

1 bin/hadoop fs -mkdir /oozie
2 bin/hadoop fs -copyFromLocal /home/shirdrn/cloud/programs/oozie-3.3.2/examples/target/oozie-examples-3.3.2-examples/examples /user/shirdrn/examples

我们拿examples中的map-reduce来进行验证,修改job.properties文件,配置内容如下所示:

1 nameNode=hdfs://m1:9000
2 jobTracker=m1:19830
3 queueName=default
4 examplesRoot=examples
5  
6 oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
7 outputDir=map-reduce

我的环境下,Namenode服务端口为hdfs://m1:9000,JobTracker为m1:19830,运行任务,执行如下命令:

1 cd /home/shirdrn/cloud/programs/oozie-3.3.2/client/target/oozie-client-3.3.2-client/oozie-client-3.3.2
2 bin/oozie job -oozie http://oozie-server:11000/oozie -config /home/shirdrn/cloud/programs/oozie-3.3.2/examples/target/oozie-examples-3.3.2-examples/examples/apps/map-reduce/job.properties -run

可以通过OozieWeb管理控制台查看提交运行的任务,如图所示:

以及,job配置,运行状态等信息,如图所示:

上面命令选项-run表示直接运行一个job,当然你可以使用其他选项,如-submit是提交job,-rerun是重新运行job,-suspend是挂起job等等,可以查看命令帮助,或参考相关文档。

参考链接

  • http://oozie.apache.org/docs/3.3.2/DG_QuickStart.html
  • http://practicalcloudcomputing.com/post/26337621577/installing-and-running-apache-oozie-3-2-x-and-possibly?543b50f0
  • http://www.cnblogs.com/cenyuhai/p/3263756.html

Oozie-3.3.2安装配置运行实践相关推荐

  1. DynaSLAM超详细安装配置运行ubantu20.0.4+opencv2.4.11+tensorflow1.4.0

    DynaSLAM超详细安装配置运行ubantu20.0.4+opencv2.4.11+tensorflow1.4.0 注:我现在ubantu的环境是基于可以运行ORB SLAM 2和ORB SLAM3 ...

  2. Ubuntu20.04安装配置运行DynaSLAM

    Ubuntu20.04安装配置运行DynaSLAM 文章目录 Ubuntu20.04安装配置运行DynaSLAM 一.安装Anaconda 二.安装依赖 三.配置Mask_RCNN环境 四.安装Dyn ...

  3. Pinpoint【环境搭建 01】JDK\HBase\Pinpoint Collector+Web 最新版 2.3.3 安装配置运行验证及脚本文件分享(避坑指南捷径指北)

    本文主要是介绍 Pinpoint 环境的部署,小伙伴儿们也可以参考 Pinpoint <官网>的<快速入门>手册,最新版本v2.3.3组件可到官方<GitHub仓库> ...

  4. Nginx编译-安装-配置-优化实践总结

    http://www.zhuxiaodong.net/2016/configure-nginx-server-support-https-http2-on-centos7/ 一些更新说明: 2016- ...

  5. vscode怎么运行android项目,windows + flutter +android+ vscode 安装配置运行流程(详细版本)...

    flutter 是由谷歌发布的一个全新的响应式.跨平台.高性能的移动开发框架,可以快速在iOS和Android上构建高质量的原生用户界面. 框架特点 快速开发:Flutter的热重载可以快速地进行测试 ...

  6. JDK 安装 配置 运行 帮助文档

    JDK 详细安装文档 --安 装 配 置 到 运 行 一.下载JDK 版本-1.8 1.下载jdk 1. 下载地址: http://www.oracle.com/technetwork/java/ja ...

  7. XMail 安装配置使用

    转自:http://blog.sina.com.cn/s/blog_66e177dd0100kwd0.html 前言 企业内部邮件系统可以用于企业内网(局域网)成员,包括人和系统之间通讯.比如一个封闭 ...

  8. XMail 安装配置使用 (1.27 )

    最近在研究xmail,找到了这份资料,在此整理收藏一笔. 原文链接 前言 企业内部邮件系统可以用于企业内网(局域网)成员,包括人和系统之间通讯.比如一个封闭的研发团队,采用代码管理系统(SVN),需要 ...

  9. 服务器搭建hue_Hue安装配置实践

    Hue是一个开源的Apache Hadoop UI系统,最早是由Cloudera Desktop演化而来,由Cloudera贡献给开源社区,它是基于Python Web框架Django实现的.通过使用 ...

最新文章

  1. SAP收货后更改采购单价格
  2. Android Hook (2) Java2java
  3. XPath和lxml类库
  4. 【计算机网络复习 物理层】2.1.1 物理层基本概念
  5. firebase登录验证_如何使用Firebase通过三步向身份验证本机添加身份验证
  6. Pandas数据帧(DataFrame)
  7. 技术研究院004---数据仓库系统的实现与使用(含OLAP重点讲解)
  8. css3制作俩面翻转盒子效果
  9. 技术要求→物理安全→防雷击
  10. kubernetes failed to start sandbox
  11. 计算机专业个人职业规划范文200字,计算机专业职业生涯规划范文
  12. AVATR阿维塔11维修手册电路图技术资料
  13. java中compar_1.java中Comparor与Comparable的问题
  14. 计算机快速看图教程,CAD快速看图教程:CAD图纸测量方法集锦
  15. syswow64删除文件_syswow64,教您syswow64是什么文件夹
  16. 苹果电脑查看python版本_Mac终端查看python版本号
  17. 春风十里不如你、与IPC的邂逅
  18. 联合分析法(Python实现)
  19. C、C++和C# 到底有什么区别了,纳闷
  20. Three.js光照贴图添加阴影(·lightMap)

热门文章

  1. c# 18位身份证源代码
  2. 基片集成波导原理_ETH研究人员开发微型红外光谱仪 可以集成到芯片当中
  3. Visual studio 2017基本配置
  4. Cloudify — 安装部署与基础使用
  5. NR 5G 协议层服务和功能
  6. MySQL学习之路:多实例无法启动排错
  7. 收集国内著名互联网公司前端/UED部门的blog,方便学习交流
  8. 在C#中使用代理的方式触发事件 (委托和事件 ) (转)
  9. 如何检测如果滚动条是否可见控件上
  10. 说说如何搭建 Nginx 反向代理 Tomcat