Oozie 集成 Hive

1) prepare shell case
$ cd ~/work/oozie-5.2.1
$ tree oozie/apps/hive2

oozie/apps/hive2
├── job.properties
├── script.q
└── workflow.xml

$ cat oozie/apps/hive2/script.q

--
DROP TABLE IF EXISTS test;
CREATE EXTERNAL TABLE test (a INT) STORED AS TEXTFILE LOCATION '${INPUT}';
insert into test values(10);
insert into test values(20);
insert into test values(30);
-- INSERT OVERWRITE DIRECTORY '${OUTPUT}' SELECT * FROM test;

$ cat oozie/apps/hive2/job.properties

nameNode=hdfs://localhost:9000
resourceManager=localhost:8032
queueName=default
jdbcURL=jdbc:hive2://localhost:10000/default
oozieRoot=user/${user.name}/oozieoozie.use.system.libpath=trueoozie.wf.application.path=${nameNode}/${oozieRoot}/apps/hive2
inputDir=data/hive2/table
outputDir=data/hive2/output

$ cat oozie/apps/hive2/workflow.xml

<workflow-app xmlns="uri:oozie:workflow:1.0" name="hive2-wf"><start to="hive2-node"/><action name="hive2-node"><hive2 xmlns="uri:oozie:hive2-action:1.0"><resource-manager>${resourceManager}</resource-manager><name-node>${nameNode}</name-node><prepare><delete path="/${oozieRoot}/${outputDir}"/><mkdir path="/${oozieRoot}/${outputDir}"/></prepare><configuration><property><name>mapred.job.queue.name</name><value>${queueName}</value></property></configuration><jdbc-url>${jdbcURL}</jdbc-url><script>script.q</script><param>INPUT=/${oozieRoot}/${inputDir}</param><param>OUTPUT=/${oozieRoot}/${outputDir}</param></hive2><ok to="end"/><error to="fail"/></action><kill name="fail"><message>Hive2 (Beeline) action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message></kill><end name="end"/>
</workflow-app>

2) upload to hdfs
$ hdfs dfs -put oozie/apps/hive2 oozie/apps/

3) run and check
firstly, guarantee HiveMetaStore and HiveServer2 have been started
$ bin/oozie job -config oozie/apps/hive2/job.properties -run
job: 0000000-220630151459153-oozie-sun_-W
$ bin/oozie job -info 0000000-220630151459153-oozie-sun_-W

Job ID : 0000000-220630151459153-oozie-sun_-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : hive2-wf
App Path      : hdfs://localhost:9000/user/sun_xo/oozie/apps/hive2
Status        : SUCCEEDED
Run           : 0
User          : sun_xo
Group         : -
Created       : 2022-06-30 09:58 GMT
Started       : 2022-06-30 09:58 GMT
Last Modified : 2022-06-30 09:59 GMT
Ended         : 2022-06-30 09:59 GMT
CoordAction ID: -Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000000-220630151459153-oozie-sun_-W@:start:                                  OK        -                      OK         -
------------------------------------------------------------------------------------------------------------------------------------
0000000-220630151459153-oozie-sun_-W@hive2-node                               OK        application_1656559415643_0018SUCCEEDED  -
------------------------------------------------------------------------------------------------------------------------------------
0000000-220630151459153-oozie-sun_-W@end                                      OK        -                      OK         -
------------------------------------------------------------------------------------------------------------------------------------

$ hdfs dfs -text "oozie/data/hive2/table/*"

10
20
30

And you can get relevant job log as following:
$ hdfs dfs -get /tmp/logs/sun_xo/logs/application_1656559415643_0018 logs/

Oozie 集成 Hive相关推荐

【阿里云EMR实战篇】以EMR测试集群版本为例，详解 Flink SQL Client 集成 Hive 使用步骤
简介: 以测试集群版本为例(EMR-4.4.1)-- Flink SQL Client 集成 Hive 使用文档作者:林志成,阿里云EMR产品团队技术支持,拥有多年开源大数据经验 1.以测试集群版本 ...
hue集成mysql报错_hue集成hive访问报database is locked
这个问题这应该是hue默认的SQLite数据库出现错误,你可以使用mysql postgresql等来替换 hue默认使用sqlite作为元数据库,不推荐在生产环境中使用.会经常出现database ...
carbondate mysql_CarbonData集成Hive、Spark
硬件准备: 系统:CentOS 7.6(1810) CPU:4核内存:16G 软件准备: 注意: 在carbondata-1.6.1中,有组件版本限制,具体为: hadoop支持到2.7.2 hiv ...
Hue（四）集成Hive
1. 说明本篇主要介绍配置Hue来使用Hive,通过Hue的Web UI来使用Hive的查询功能.配置分为两部分,一个是针对Hive本身的配置,一个是针对Hue的配置. 2. 配置Hive Hive ...
Apache Hue: Hue集成Hive
Hue集成Hive 如果需要配置hue与hive的集成,我们需要启动hive的metastore服务以及hiveserver2服务(impala需要hive的metastore服务,hue需要hvie ...
hue的安装与集成hive使用
hue官网: http://gethue.com hue cdh版本下载地址: http://archive.cloudera.com/cdh5/cdh/5/ 搜索 hue-3.9.0-cdh5.1 ...
Flink 1.10 正式发布！——与Blink集成完成，集成Hive，K8S
Apache Flink社区宣布Flink 1.10.0正式发布! 本次Release版本修复1.2K个问题,对Flink作业的整体性能和稳定性做了重大改进,同时增加了对K8S,Python的支持. ...
spark指定hive字段_Spark2.0集成Hive操作的相关配置与注意事项
前言补充说明 Hive中metastore(元数据存储)的三种方式:内嵌Derby方式 Local方式 Remote方式 [一].内嵌Derby方式这个是Hive默认的启动模式,一般用于单元测试,这 ...
Hue 集成Hive
如果需要配置hue与hive的集成,我们需要启动hive的metastore服务以及hiveserver2服务(impala需要hive的metastore服务,hue需要hvie的hiveserve ...

Oozie 集成 Hive

Oozie 集成 Hive相关推荐

最新文章

热门文章