Hive Merge详解

参考

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Merge

https://community.cloudera.com/t5/Community-Articles/Hive-ACID-Merge-by-Example/ta-p/245402

说明

Hive在2.2版本之后开始支持Merge操作，并且Merge只能在支持ACID的表上执行

语法

MERGE INTO <target table> AS T USING <source expression/table> AS S
ON <boolean expression1>
WHEN MATCHED [AND <boolean expression2>] THEN UPDATE SET <set clause list>
WHEN MATCHED [AND <boolean expression3>] THEN DELETE
WHEN NOT MATCHED [AND <boolean expression4>] THEN INSERT VALUES<value list>

建表语句

CREATE DATABASE merge_data;CREATE TABLE merge_data.transactions(ID int,TranValue string,last_update_user string)
PARTITIONED BY (tran_date string)
CLUSTERED BY (ID) into 5 buckets
STORED AS ORC TBLPROPERTIES ('transactional'='true');CREATE TABLE merge_data.merge_source(ID int,TranValue string,tran_date string)
STORED AS ORC;

导入数据

INSERT INTO merge_data.merge_source VALUES
(1, 'value_01', '20170410'),
(4, NULL, '20170410'),
(7, 'value_77777', '20170413'),
(8, NULL, '20170413'),
(8, 'value_08', '20170415'),
(11, 'value_11', '20170415');INSERT INTO merge_data.transactions PARTITION (tran_date) VALUES
(1, 'value_01', 'creation', '20170410'),
(2, 'value_02', 'creation', '20170410'),
(3, 'value_03', 'creation', '20170410'),
(4, 'value_04', 'creation', '20170410'),
(5, 'value_05', 'creation', '20170413'),
(6, 'value_06', 'creation', '20170413'),
(7, 'value_07', 'creation', '20170413'),
(8, 'value_08', 'creation', '20170413'),
(9, 'value_09', 'creation', '20170413'),
(10, 'value_10','creation', '20170413');

在导入目标表数据会遇到以下问题

错误一：

FAILED: SemanticException [Error 10096]: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict

set hive.exec.dynamic.partition.mode=nonstrict;

错误二：

FAILED: SemanticException [Error 10265]: This command is not allowed on an ACID table merge_data.transactions with a non-ACID transaction manager. Failed command: INSERT INTO merge_data.transactions PARTITION (tran_date) VALUES

set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

错误三：

FAILED: RuntimeException [Error 10264]: To use DbTxnManager you must set hive.support.concurrency=true
Exception in thread "main" java.lang.NullPointerException
   at org.apache.hadoop.hive.ql.Driver.releaseLocksAndCommitOrRollback(Driver.java:1186)
   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1324)
   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
   at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
   at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
   at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

set hive.support.concurrency=true;

修改完以上三个参数之后，导入数据成功

Merge操作

MERGE INTO merge_data.transactions AS T
USING merge_data.merge_source AS S
ON T.ID = S.ID and T.tran_date = S.tran_date
WHEN MATCHED AND (T.TranValue != S.TranValue AND S.TranValue IS NOT NULL) THEN UPDATE SETTranValue = S.TranValue,last_update_user = 'merge_update'
WHEN MATCHED AND S.TranValue IS NULL THEN DELETE
WHEN NOT MATCHED THEN INSERT VALUES (S.ID, S.TranValue, 'merge_insert', S.tran_date
);

查看结果

至此，我们就完美实现了Hive的Merge操作！

扩展delete操作

既然我们已经是事务表了那让我们试一试删除操作

从临时表导入数据

insert into table merge_data.transactions PARTITION (tran_date)
select ID,TranValue,last_update_user,tran_date from merge_data.transactions_tmp
cluster by id;

cluster by (id) - 指定getPartition以哪个字段来进行hash散列，并且排序字段也是指定的字段，默认以正序进行排序
distribute by(uid) – 指定getPartition以哪个字段来进行hash散列
sort by(uid asc) – 指定排序字段，以及排序规则

Hive Merge详解相关推荐

Hive beeline详解
Hive客户端工具后续将使用Beeline 替代HiveCLI ,并且后续版本也会废弃掉HiveCLI 客户端工具,Beeline是 Hive 0.11版本引入的新命令行客户端工具,它是基于SQLLi ...
Hive入门详解操作
Hive 第一章 Hive简介 1.1. Hive的简介 1.1.1 hive出现的原因 FaceBook网站每天产生海量的结构化日志数据,为了对这些数据进行管理,并且因为机器学习的需求,产生了hiv ...
Hive Metastore详解大全
1.介绍 Hive所有表和分区的元数据都通过Hive Metastore进行访问.使用JPOX ORM解决方案(Data Nucleus)持久化元数据,因此它支持的任何数据库都可以被Hive使用.它支 ...
java调用kettle连hive_使用java连接hive，并执行hive语句详解
安装hadoop 和 hive我就不多说了,网上太多文章自己看去首先,在机器上打开hiveservice hive --service hiveserver -p 50000 & 打开50 ...
【OpenCV3】图像通道分离与合并——cv::split()与cv::merge()详解
在图像处理中,尤其是处理多通道图像时,有时需要对各个通道进行分离,分别处理:有时还需要对分离处理后的各个通道进行合并,重新合并成一个多通道的图像.opencv2和opencv3中实现图像通道的合并与分 ...
delete hive_「挑战30万年薪」 Hive语句详解之从SQL到HQL的应转习惯
Hive不支持等值连接 SQL中对两表内联可以写成: select * from dual a,dual b where a.key=b.key; Hive中应为: select * from dua ...
大数据技术-hive窗口函数详解
有不少同学一听这个标题,hive窗口函数是什么鬼?没听说过还有窗口函数这个东西啊,其实它的用处可大了,下面听小千慢慢道来. hive窗口函数窗口函数指定了函数工作的数据窗口大小(当前行的上下多少行) ...
java hive 查询语句,使用java连接hive，并执行hive语句详解
packageasia.wildfire.hive.service; importjava.sql.*; importjava.sql.Date; importjava.text.SimpleDate ...
大数据开发笔记（四）：Hive分区详解
✨大数据开发笔记推荐: 大数据开发面试知识点总结_GoAI的博客-CSDN博客_大数据开发面试本文详细介绍大数据hadoop生态圈各部分知识,包括不限于hdfs.yarn.mapreduce.h ...